Dynamic scaling of a cluster of computing nodes

ABSTRACT

Techniques are described for managing distributed execution of programs, including by dynamically scaling a cluster of multiple computing nodes performing ongoing distributed execution of a program, such as to increase and/or decrease computing node quantity. An architecture may be used that has core nodes that each participate in a distributed storage system for the distributed program execution, and that has one or more other auxiliary nodes that do not participate in the distributed storage system. Furthermore, as part of performing the dynamic scaling of a cluster, computing nodes that are only temporarily available may be selected and used, such as computing nodes that might be removed from the cluster during the ongoing program execution to be put to other uses and that may also be available for a different fee (e.g., a lower fee) than other computing nodes that are available throughout the ongoing use of the cluster.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,such as with the computing systems being co-located (e.g., as part of alocal network) or instead located in multiple distinct geographicallocations (e.g., connected via one or more private or sharedintermediate networks). Such groups of interconnected computing systemsare increasingly useful for various reasons, including to supportincreasingly large software programs and input data sets that may bedistributed across multiple computing systems.

Various approaches have been taken to providing groups of interconnectedcomputing systems. For example, data centers housing significant numbersof interconnected co-located computing systems have become commonplace,such as private data centers that are operated by and on behalf of asingle organization, and public data centers that are operated byentities as businesses to provide computing resources to customers. Somepublic data center operators provide network access, power, and secureinstallation facilities for hardware owned by various customers, whileother public data center operators provide “full service” facilitiesthat also include hardware resources made available for use by theircustomers. However, as the scale and scope of typical data centers andcomputer networks has increased, the task of provisioning,administering, and managing the associated physical computing resourceshas become increasingly complicated.

The advent of virtualization technologies for commodity hardware hasprovided some benefits with respect to managing large-scale computingresources for many customers with diverse needs, allowing variouscomputing resources to be efficiently and securely shared by multiplecustomers. For example, virtualization technologies such as thoseprovided by VMWare, XEN, Linux's KVM (“Kernel-based Virtual Machine”),or User-Mode Linux may allow a single physical computing machine to beshared among multiple users by providing each user with one or morevirtual machines hosted by the single physical computing machine, witheach such virtual machine being a software simulation acting as adistinct logical computing system that provides users with the illusionthat they are the sole operators and administrators of a given hardwarecomputing resource, while also providing application isolation andsecurity among the various virtual machines.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are network diagrams illustrating example embodiments ofuse of a distributed program execution service by remote users.

FIGS. 2A-2E illustrate examples of managing the monitoring and modifyingof distributed program execution on multiple computing nodes.

FIG. 3 is a block diagram illustrating an example embodiment of acomputing system for managing distributed program execution.

FIGS. 4A-4C illustrate a flow diagram of an example embodiment of aDistributed Program Execution Service System Manager routine.

FIG. 5 illustrates a flow diagram of an example embodiment of a routinefor obtaining user-specified program execution configurationinformation.

FIG. 6 illustrates a flow diagram of an example embodiment of a routinefor performing distributed program execution.

FIG. 7 illustrates a flow diagram of an example embodiment of aComputing Node Manager routine.

FIG. 8 illustrates a flow diagram of an example embodiment of a CapacityUsage Manager routine.

DETAILED DESCRIPTION

Techniques are described for managing distributed execution of programs.In at least some embodiments, the techniques include dynamically scalinga cluster of multiple computing nodes that are used to perform theongoing distributed execution of a particular program, such as toincrease and/or decrease the quantity of computing nodes in the clusterat various times and for various reasons during ongoing use of thecluster. In addition, in at least some embodiments, the managing of thedistributed execution of a program on a cluster of multiple nodesincludes using an architecture for the multiple computing nodes of thecluster that facilitates the dynamic scaling, including by having atleast some of the computing nodes act as core nodes that eachparticipate in a distributed storage system for the distributed programexecution, and optionally having one or more other computing nodes thatact as auxiliary nodes that do not participate in the distributedstorage system. Furthermore, as part of performing the dynamic scalingof a cluster, the techniques may in some embodiments and situationsselect and use computing nodes that are only temporarily available, suchas computing nodes that might be removed from the cluster during theongoing program execution to be put to other uses (e.g., otherpreexisting uses to which those computing nodes were already allocatedbefore their use in the cluster), but that may also be available for adifferent fee (e.g., a lower fee) than other computing nodes that areavailable throughout the ongoing use of the cluster. To perform thedistributed execution of the program on the cluster, the program may beseparated into multiple execution jobs that are each executed on one ofthe computing nodes of the cluster, such as by executing one or more ofthe execution jobs on each auxiliary node of the cluster and optionallyon some or all of the core nodes of the cluster, and the architecturefor the distributed program execution may further include one or more ofthe cluster computing nodes that act as administrative computing nodesto control or monitor at least some aspects of the distributed executionof the program on the other core and auxiliary computing nodes of thecluster. Additional details related to managing the distributedexecution of programs are included below.

In at least some embodiments, some or all of the techniques may beautomatically performed by a Distributed Program Execution (“DPE”)service, as described in greater detail below, such as a service that isprovided by a corresponding automated system and that executes multipleprograms on behalf of multiple customers or other clients of theservice. Moreover, in addition to managing distributed execution ofprograms, such a DPE service may in some embodiments provide variouscomputing nodes having access to various computing resources (e.g.,local memory, local storage, local CPU cycles, networks betweencomputing nodes and to external computing systems, remote storage, etc.)that are available for use in executing programs for clients in adistributed manner. The computing nodes used as part of a cluster forthe distributed execution of a program may have various forms in variousembodiments (e.g., may each be a physical computing system and/or avirtual machine that is hosted on one or more physical computingsystems), and may each have access to various computing resources. Asdescribed in greater detail below, the computing resources used toexecute a particular program may include particular quantities ofmultiple types of computing resources (e.g., an aggregate amount of RAMor other memory of one or more physical computing systems that is used,such as by one or more computing nodes; an aggregate amount of networkbandwidth capacity used on one or more networks, such as by one or morecomputing nodes; an aggregate amount of storage space used on one ormore storage devices, such as by one or more computing nodes, andoptionally by using local storage devices of multiple computing nodes;etc.). Furthermore, the computing nodes that the DPE service uses may insome embodiments be of multiple resource configuration types that may beselected and used in different situations, with each resourceconfiguration type having a distinct combination of associated computingresources for computing nodes of that type, as discussed in greaterdetail below. Moreover, the computing nodes that the DPE service usesmay in some embodiments be available from multiple different sources,with at least one source providing computing nodes that are onlytemporarily available for use in a cluster (e.g., computing nodes whoseavailability for ongoing use in the cluster is not guaranteed, such thatthe computing nodes might be removed from the cluster during the ongoingprogram execution to be put to other uses, but also may remain availablefor use in the cluster until the cluster use ends if the other uses ofthe computing nodes do not occur), as discussed in greater detail below.

As previously noted, a cluster for use in the distributed execution of aprogram may in at least some embodiments include multiple core computingnodes that participate in a distributed storage system for use by thecluster, such as to store input data used in the distributed programexecution and/or output data generated by the distributed programexecution. The distributed storage system may have various forms invarious embodiments, such as a distributed file system, a distributeddatabase, etc., and in some embodiments may provide various mechanismsto enhance data availability (e.g., by storing multiple copies of atleast some groups of data, such as to enhance the likelihood that atleast one copy of a particular group of data remains available if a corecomputing node storing another copy of that data group fails orotherwise becomes unavailable). When executing a particular executionjob for a program on a computing node of a cluster for that program,information related to the execution may be accessed and used in variousmanners. For example, when the execution of the execution job isinitiated on a computing node, the input data to be used by theexecution job may be locally stored on the computing node (e.g., on alocal hard disk or other local storage device) to facilitate access tothat input data during execution, and any software instructions to beexecuted for the execution job may similarly be locally stored on thecomputing node. If the computing node on which the execution occurs is acore computing node that already locally stores the input data to beused as part of its participation in the distributed storage system(e.g., if that core computing node is selected to execute the executionjob because it already locally stores that input data on a storagedevice of the core node), the execution may merely include retrievingand using that input data from the distributed storage system as needed,and storing any generated results in the local portion of thedistributed storage system. Alternatively, if the computing node onwhich the execution occurs is a core node that does not already locallystore the input data to be used or is an auxiliary computing node notparticipating in the distributed storage system, the computing node mayretrieve that input data from one or more of the core nodes that dostore that input data, locally store that retrieved data temporarily onthe computing node, and optionally later send any generated results datato one or more core nodes for storage in the distributed storage system.Furthermore, in at least some embodiments, the input data and/orgenerated output data in the distributed storage system may further beexchanged with or otherwise stored in one or more remote storagelocations, and in some embodiments such remote storage may be used inlieu of a local distributed storage system, as discussed in greaterdetail below.

The use of an architecture having core computing nodes and auxiliarycomputing nodes in a cluster may facilitate the dynamic scaling of thecomputing nodes for that cluster in various manners in variousembodiments. For example, if a particular computing node will remain ina cluster until the use of the cluster ends (e.g., until the distributedexecution of one or more programs by the cluster is completed), it ispreferable in at least some embodiments to use that computing node as acore computing node participating in a distributed storage system forthe cluster, as doing so increases the aggregate size of the distributedstorage system, and may also allow that core computing node to moreefficiently perform the execution of execution jobs for which it alreadylocally stores associated input data. Alternatively, if a particularcomputing node will be removed from a cluster before the use of thecluster ends (whether based on a decision by or on behalf of a clientassociated with the cluster to remove the computing node as part ofscaling the cluster, or instead based on a potential or actual removalof a temporarily available computing node in the cluster that occursindependently of the client to enable the computing node to be put toother uses), it is preferable in at least some embodiments to use thatcomputing node as an auxiliary computing node not participating in thedistributed storage system for the cluster, as removing a local portionof the distributed storage system that would have been stored on thatauxiliary node if it had instead been a core node while the cluster isin use may cause one or more of the copies of data stored in that localportion to be lost (e.g., if other copies of the data are notavailable), may cause additional work in recreating other copies (ifpossible) of the data stored in the local portion, and/or in somesituations may cause some or all of the aggregate functionality of theentire distributed storage system to be lost (e.g., if the structure ofthe distributed storage system does not support the removal of aconstituent computing node while the distributed storage system is inuse).

Accordingly, core computing nodes and auxiliary computing nodes may beused in various manners in various clusters. When a cluster is created,some or all of the computing nodes in the cluster may initially beselected to be core computing nodes, and optionally zero or more of thecomputing nodes in the cluster may initially be selected to be auxiliarycomputing nodes. Similarly, when a cluster is dynamically modified whileit is in use, a particular one of core computing nodes and auxiliarycomputing nodes may be selected as a type of computing node to bemodified, such as to increase the quantity of computing nodes in thecluster by adding one or computing nodes of the selected type and/or toreduce the quantity of computing nodes in the cluster by removing one ormore computing nodes of the selected type. If the quantity of computingnodes in a cluster is dynamically reduced while the cluster is in use,auxiliary computing nodes may be the only computing nodes that areselected for removal in some embodiments, while in other embodimentsauxiliary computing nodes may preferably be selected for removal if theyare available but with core computing nodes being removed if sufficientauxiliary computing nodes are not available in the cluster. If thequantity of computing nodes in a cluster is dynamically increased whilethe cluster is in use, one or both of the core and auxiliary computingnode types may be selected for addition in at least some embodiments,such as based at least in part on the situation surrounding the increase(e.g., if the increase in the quantity is intended to be permanent untiluse of the cluster ends, then add core nodes, and otherwise add some orall of the additional computing nodes as auxiliary nodes if the increasemay be temporary). In some embodiments and situations, a particularcluster may include only core computing nodes or only auxiliarycomputing nodes. In addition, when the quantity of computing nodes in acluster is being modified as part of dynamically scaling the cluster,temporarily available computing nodes may in some embodiments preferablybe selected if they are available for one or both activities of addingand removing computing nodes, while in some embodiments temporarilyavailable computing nodes may preferably not be selected if othercomputing nodes are available for one or both activities of adding andremoving computing nodes, such as based on a current price for using thetemporarily available computing nodes and/or on one or more otherfactors.

Situations in which a cluster of computing nodes may be dynamicallymodified while in use include the following non-exclusive examples. As afirst example, the distributed execution of a program may use at least aminimum quantity of computing nodes at all times, and may occasionallyuse additional computing nodes (e.g., to temporarily increase the totalquantity to handle temporary increases in computing load)—if so, thecluster may be created at a first time with a quantity of core computingnodes that corresponds to the minimum quantity, and with any othercomputing nodes that are temporarily added or subsequently removed at alater second time being auxiliary computing nodes. In such a situation,if the minimum quantity of computing nodes for the cluster is increasedat a later second time while the cluster is in use, the new nodes addedto reach the new higher minimum quantity may be selected to beadditional core computing nodes. As a second example, the distributedexecution of a program may have multiple distinct stages or phases, anddifferent amounts of computing resources may be desired for thedifferent stages or phases—if so, and if a later stage/phase is desiredto have a lower quantity of computing nodes than an earlier stage/phase,the cluster may be initially created at a first time with a quantity ofcore computing nodes that corresponds to the lowest total quantity ofcluster nodes that are expected for any of the stages/phases, and withany other computing nodes that are initially included at the first timeor later added or removed at a second time being auxiliary computingnodes. As a third example, the distributed execution of a program maybegin at a first time with a minimum quantity of computing nodes toachieve a desired goal (e.g., to be expected to complete the distributedprogram execution before a specified time), but may dynamically decideat a later second time to increase the quantity of computing nodes inthe cluster during ongoing program execution based on current conditionsat the second time (e.g., based on a cost of the additional computingnodes at the second time being sufficiently low, such as if temporarilyavailable computing nodes are available at a price below a specifiedthreshold; based on the status of the distributed program execution atthe second time, such as having not reached a desired degree ofcompletion; based on a particular specified second time being reachedwhile the distributed program execution is ongoing; etc.)—if so, thecluster may be created with a quantity of core computing nodes thatcorresponds to the minimum quantity, and with any other computing nodesthat are added at the later second time being selected to be core orauxiliary computing nodes depending on the planned permanence ortemporariness, respectively, of the added nodes. It will be appreciatedthat a cluster may be dynamically modified while in use in various othersituations and in various other manners.

As previously noted, the computing nodes that are available to be usedin a cluster may in some embodiments and situations include multipletypes of computing nodes having distinct computing resourceconfigurations. If so, the different computing resource configurationcomputing node types may optionally be used as part of the dynamicscaling of the cluster in various manners in various embodiments. Forexample, in some such embodiments, all of the core computing nodes maybe of a single resource configuration computing node type and all of theauxiliary computing nodes may be of a single resource configurationcomputing node type, but with the resource configuration computing nodetypes being different for the core and auxiliary nodes—such aconfiguration may be useful if the different uses for the nodes involvedifferent computing resource configurations (e.g., if the resourceconfiguration computing node type for the core computing nodes includesat least a minimum amount of local storage for use within thedistributed storage system, while the resource configuration computingnode type for the auxiliary computing nodes includes at least a minimumamount of local memory and/or CPU cycles to facilitate the execution ofexecution jobs by the auxiliary nodes). In other embodiments, one orboth of the core and auxiliary computing nodes may include computingnodes of multiple different resource configuration computing node types.In addition, the resource configuration computing node types may befurther used as part of dynamic scaling of a cluster while the clusteris in use, such as to specify to add or to remove computing nodes of aspecified resource configuration computing node type from the coreand/or auxiliary computing nodes in the cluster (e.g., to add or removea specified quantity of computing nodes of the specified resourceconfiguration computing node type, to remove all computing nodes of thespecified resource configuration computing node type from at least oneof the core and auxiliary computing nodes, etc.). Computing nodes ofdifferent resource configuration computing node types may also haveother differing attributes that are used to affect their selection anduse in at least some embodiments, including an associated cost of use,as discussed in greater detail elsewhere.

In addition, the computing nodes that are available to be used in acluster may in some embodiments and situations include computing nodesfrom different sources that have different characteristics (e.g.,associated price for use, associated use conditions, etc.). For example,the multiple sources may include a first source of temporarily availablecomputing nodes whose associated use conditions include one or more useconditions related to the circumstances of removing such computing nodesfrom ongoing use in a cluster due to a distinct planned use for thecomputing nodes, and may include at least one second source of computingnodes that are not similarly temporarily available. If multiple suchsources of computing nodes are available, computing nodes from differentsources may optionally be used as part of the dynamic scaling of thecluster in various manners in various embodiments. For example, in somesuch embodiments, all of the core computing nodes may be from a singlesource and all of the auxiliary computing nodes may be from a differentsingle source. In other embodiments, one or both of the core andauxiliary computing nodes may include computing nodes from multipledifferent sources. In addition, the differing computing node sources maybe further used as part of dynamic scaling of a cluster while thecluster is in use, such as to specify to add or to remove computingnodes from a specified source for the core and/or auxiliary computingnodes in the cluster (e.g., to add or remove a specified quantity ofcomputing nodes from the specified source, to remove all computing nodesfrom the specified source from at least one of the core and auxiliarycomputing nodes, etc.).

Some or all of the temporarily available computing nodes may in someembodiments be allocated for one or more particular uses other than inclients' clusters, but may currently not be used for those otherparticular uses, such as if the temporarily available computing nodesrepresent excess execution or computing capacity for those otherparticular uses at a current time. As previously noted, such excess orotherwise unused execution capacity of a pool or other group of one ormore temporarily available computing nodes may be made available toexecute programs in clusters on behalf of clients on a temporary ornon-guaranteed basis, such that the excess program execution capacitymay be available to execute such programs until a time that the programexecution capacity is desired for other purposes (e.g., for preferentialor reserved use by one or more other users). If a cluster does notinclude sufficient computing nodes of a type other than temporarilyavailable computing nodes to maintain the ongoing distributed executionof a program, and those temporarily available computing nodes areremoved from the cluster to be put to other uses, the execution of theprogram by the cluster may be automatically terminated (e.g., shut down,aborted, etc.) by the DPE service in at least some such situations.

In some embodiments, at least some computing nodes that would otherwisebe available for use by the DPE service in any client's cluster may beallocated to one or more clients for preferential use by those clients,such that each of those clients has priority access relative to otherclients to use a respective quantity of those computing nodes. In onesuch embodiment, the priority access of the clients may be based on eachof the clients having dedicated or exclusive access to use a particularquantity of those computing nodes (or otherwise having a respectiveamount of program execution capacity), such as by having one or morededicated computing nodes and/or portions thereof that are allocated forreserved or guaranteed use by the client in executing programs. Forexample, in at least some such embodiments, a respective quantity ofsuch computing nodes may be allocated to a particular client fordedicated access over a specified period of time, such as in a manneranalogous to a lease of one or more physical computing systems so that acorresponding amount of program execution capacity is available forexecuting programs on behalf of the client throughout the specifiedperiod of time. In addition, in some embodiments, a client may be givenpreferential or other dedicated access to execute programs based on oneor more factors, such as fees paid by the client, an agreement to acontractual obligation for using the dedicated access for a period oftime and/or subject to other conditions, etc.

In at least some embodiments, at least some of the computing nodes thatare allocated to one or more clients for dedicated or other preferentialuse may be used as excess program execution capacity during some times,such as when the allocated capacity is not being used for a preferentialuse and/or when the client to whom the capacity is allocated makes theallocated capacity available for use by other clients. In this manner,at least some computing nodes that are allocated to a first group ofclients may become available from time to time to temporarily executeprograms on behalf of other clients. When the demand for excesscomputing capacity of such temporarily available computing nodes frommultiple clients is greater than the currently available supply,particular clients and programs may be selected to use the excesscapacity in various ways. For example, in some embodiments, a client whorequests excess computing capacity to execute a program may be givenpriority over other such clients based on one or more factors, such asfees offered and/or paid by the client, a time that a request to use theexcess capacity is submitted, a time at which or by which excesscapacity use is requested to start, an amount of time or a deadline bywhich excess capacity use is requested to be completed, a type and/oramount of excess capacity requested (e.g., a specified quantity ofcomputing nodes), etc. Additional details related to the use ofdedicated program execution capacity as excess program executioncapacity are included below.

In addition, in some embodiments, the DPE service may further includeon-demand variable program execution capacity from particular computingnodes that is available to satisfy dynamically received requests ofclients to execute programs (e.g., immediately upon request of thoseclients, at an indicated future time, at some time during an indicatedfuture time period, etc.), such that the one or more programs indicatedby such a request may be executed if computing resources sufficient tosatisfy the requested execution are available at (or near) the requestedtime, but without such a request being guaranteed to be satisfied. Forexample, after a variable program execution capacity request is receivedto dynamically execute one or more programs on behalf of a client (e.g.,for immediate execution), the one or more programs may be executed forthe client if an appropriate amount of program execution capacity iscurrently available for executing those programs (e.g., if it isdetermined that there is a sufficient number of available computingnodes with resources to satisfy the request), and otherwise the requestmay be denied (or in some cases, postponed). Thus, in some suchembodiments, such a request for variable program execution capacity maybe unsuccessful, such as if the appropriate amount of program executioncapacity is not available to execute the one or more programs at thetime of the requested execution (e.g., if it is determined that thereare not enough currently available computing nodes with sufficientresources to satisfy a request for immediate execution).

In some embodiments, the computing nodes used to provide the on-demandvariable program execution capacity may be distinct from the computingnodes used to provide dedicated program execution capacity and/or fromthe computing nodes used to provide excess program executioncapacity—thus, if some of the computing nodes used to provide theon-demand variable program execution capacity are not in use, in someembodiments they may be used to provide excess program executioncapacity until on-demand variable program execution capacity requestsare received, while in other embodiments they may not be used to provideexcess program execution capacity. In addition, in some embodiments,after a variable program execution capacity request for immediate (orscheduled) execution is satisfied and successfully initiates executionof one or more programs on behalf of a respective client, the ongoinguse of the variable program execution capacity may be guaranteed tocontinue until some future time, such as a time of the respectiveclient's choosing (e.g., based on an indication received from therespective client to terminate use of and/or release the programexecution capacity) and/or a time based upon design by the client (e.g.,the one or more programs executing on behalf of the client automaticallyterminate the execution, such as upon completion of a task or otherwise;an application and/or other client interacting with the one or moreprograms executing on behalf of the respective client terminates theexecution, such as if the one or more programs executing for therespective client are provided for interactive use of others; etc.).Alternatively, in some embodiments, the guaranteed ongoing use of thevariable program execution capacity may be subject to certainlimitations, such as to be guaranteed that the DPE service will notpreempt the use for other purposes, but not to be guaranteed againstfailure or other unavailability of the computing nodes executing theprograms.

In addition, as previously noted, in at least some embodiments dedicatedprogram execution capacity functionality of a DPE service may beavailable for one or more clients, such that each of the one or moreclients may have some dedicated amount of program execution capacitythat is allocated by the DPE service and guaranteed to be available forexecuting programs on behalf of the one or more clients (e.g., adedicated number of one or more computing nodes provided by the DPEservice, a dedicated portion of the computing resources provided by oneor more of the computing nodes of the DPE service, etc.). For example,the guaranteed availability of the dedicated program execution capacitymay be provided in conjunction with one or more QoS (“Quality OfService”) criteria, such as that the dedicated program executioncapacity will be available within a specified period of time after arequest to initiate execution of an indicated program (e.g., a period oftime that corresponds to terminating execution of other programs usingexcess computing capacity and to initiating execution of the indicatedprogram). In contrast to the variable program execution capacityfunctionality and excess program execution capacity functionality, aclient of dedicated-capacity functionality may always have somespecified amount of program execution capacity available for executingprograms on behalf of the client during a specified use period, barringunforeseen and/or occasional events rendering services of the DPEservice unavailable (e.g., service failure, maintenance, cancellation,etc).

The determination of whether and how to perform dynamic scaling for acluster, including to add and/or remove core or auxiliary computingnodes that are optionally of a particular resource configurationcomputing node type and/or optionally from a particular source, may bemade in various manners in various embodiments. For example, in someembodiments, a user who is a client of the DPE service may interactivelyspecify various information for use in the distributed execution of anindicated program, such as via a GUI (graphical user interface) of theDPE service. Such user-specified information may include instructions toperform one or more of the following non-exclusive list: to start acluster at the current time (or at a different specified future time)with an indicated quantity of computing nodes (optionally with anindication of how many, if any, of the computing nodes are to beauxiliary computing nodes, although in some embodiments the use ofauxiliary computing nodes may not be indicated to some or all clients);to dynamically modify a cluster at the current time (or at a differentspecified future time) by adding and/or removing an indicated quantityof computing nodes (optionally with an indication of whether thecomputing nodes are to be auxiliary computing nodes and/or corecomputing nodes, although in some embodiments the use of auxiliarycomputing nodes may not be indicated to some or all clients); and toperform other operations for a cluster (e.g., to suspend some or all ofthe execution, to resume previously suspended execution, etc.); to useone or more computing nodes of a specified resource configuration typein a specified manner, including to add and/or remove computing nodes ofthat type for a particular cluster; to use one or more computing nodesfrom a specified source in a specified manner, including to add and/orremove computing nodes from that source for a particular cluster; etc.In addition, in some embodiments, one or more scaling-related criteriamay be defined for a cluster, whether by an associated client or by theDPE service, that specify particular actions to be performed ifspecified conditions are satisfied, including to dynamically modify acluster by adding and/or removing an indicated quantity of computingnodes (optionally with an indication of whether the computing nodes areto be auxiliary computing nodes and/or core computing nodes). If a userinstruction or defined scaling criteria specifies to add or remove oneor more computing nodes without indicating whether the modificationapplies to core and/or auxiliary computing nodes, the DPE service may inat least some embodiments automatically select which type of computingnodes to use for the modification, such as by using rules or factorsdescribed in greater detail elsewhere. As is also discussed in greaterdetail elsewhere, the user-specified instructions and/or defined scalingcriteria may in some embodiments further specify particular resourceconfiguration computing node types and/or particular computing nodesources to be used in various manners when adding or removing computingnodes. The defined scaling criteria may have various forms in variousmanners, including in some embodiments to be provided or analyzed by theDPE service by executing software instructions (e.g., as part of aprogram, applet, etc.) provided by a client.

In order to determine if one or more defined scaling criteria for thedistributed execution of a program on a cluster are satisfied, as wellas to otherwise track the distributed program execution, the DPE servicemay in some embodiments dynamically monitor the distributed programexecution for some or all clusters. Such dynamic monitoring may include,for example, determining the status of execution of the program on eachof the multiple computing nodes of a cluster, and may be performed invarious manners in various embodiments. For example, in someembodiments, each cluster may include at least one administrative nodeselected to act as a master computing node for the cluster, and themaster node may gather some types of status information regarding theongoing distributed program execution by the cluster, such as from theexecution jobs executing on other cluster computing nodes (e.g., a stageor phase of execution of an execution job, such as with respect toparticular operations that are performed; an estimate of the amount ofexecution that has been performed and/or is remaining, such as measuredby time or a percentage completeness; etc.), and if so the DPE servicemay automatically obtain some or all of that status information from themaster node. In other embodiments, the DPE service may automaticallygather other types of status information, such as directly fromexecution jobs executing on the cluster computing nodes, by interactingwith manager modules of the DPE service that are local to various of thecluster computing nodes to determine status information for thatcomputing node, etc. Furthermore, in at least some embodiments, thedynamic monitoring may include automatically gathering information aboutvarious types of resource usage by the distributed execution of aprogram on a cluster, such as resource usage specific to one or moreparticular cluster computing nodes that execute the program and/oraggregate resource usage from the distributed execution of the programacross the entire cluster of computing nodes. The types of resourceusage that are monitored and measured may vary in various embodiments,such as to include one or more of the following non-exclusive list, andmay further be measured in absolute terms (e.g., a measured quantity ofuse) and/or in relative terms (e.g., a percentage used of the totalavailable, to reflect the difference between an amount used by oneprogram and the amount used by another program, etc.): network bandwidthcapacity used, such as between two or more computing nodes of thecluster or more generally for any network resources used by the DPEservice; disk I/O (“input/output”) capacity used for a particularcluster computing node and/or for a particular hard disk or otherstorage device used by one or more cluster computing nodes; amount ofvolatile or non-volatile memory used; amount of non-volatile storageused; amount of CPU cycles used or other measure of CPU utilization; busbandwidth capacity specific to a particular cluster computing node; etc.In addition, in at least some embodiments, the dynamic monitoring mayfurther include automatically determining whether the execution of oneor more programs by the DPE service is contributing to a bottleneck withrespect to access of one or more types of computing resources for otherprograms, such as for other programs of the DPE service. Such bottleneckdetection may be performed in various manners, including by detectingwhen usage of particular computing resources rises above predefinedthresholds, by detecting when performance of one or more other programsare having their execution delayed or otherwise hindered due to lack ofavailable computing resources, etc. Additional details related todynamically monitoring the ongoing distributed execution of a program ona cluster of multiple computing nodes are included below.

In addition, the dynamic modifying of the ongoing distributed executionof a program on a cluster of multiple computing nodes may be performedin various manners in various embodiments, including in some embodimentsand situations based on information obtained from dynamic monitoringactivities (e.g., to reflect defined scaling criteria that aresatisfied, to reduce or eliminate bottlenecks, etc.). As previouslynoted, in some embodiments and for some types of programs, the dynamicmodifying may include automatically scaling a cluster being used fordistributed execution of a program while the distributed execution isongoing, such as to expand the cluster during ongoing execution byadding one or more additional computing nodes and/or to shrink thecluster during ongoing execution by removing one or more of thecomputing nodes from the cluster. Cluster expansion may be performed,for example, to enable program execution to complete sooner, such as ifexecution on one or more cluster computing nodes is taking longer thanexpected, if execution of the program is being hindered by lack ofsufficient computing resources and the additional computing nodes willprovide access to additional computing resources that were lacking, if amaster node or other cluster computing node has failed or otherwisebecome unavailable and the additional computing node(s) are configuredto automatically take the place of the unavailable computing nodes, ifthe cost of using additional computing nodes is sufficiently low (e.g.,for temporarily available computing nodes, such as if their currentassociated price for use is below a defined threshold), etc. Clustershrinking may be performed, for example, to more efficiently useresources, such as if the distributed program execution is progressingfaster than expected, if one or more cluster computing nodes are usingtoo many computing resources and those computing nodes are shut down tothrottle the excess computing resource usage, if one or more clustercomputing nodes are not currently being used (e.g., have completed theirportion of the distributed program execution and are removed from thecluster so as to be available for other uses and/or to prevent theongoing distributed program execution from being responsible for ongoingfees for the computing node if it was part of the cluster), to removeall computing nodes from a cluster if a sufficient subset of the clustercomputing nodes are not available for the ongoing execution (e.g.,during an initial phase of the distributed program execution if aminimum percentage of cluster computing nodes and/or a minimum absolutequantity of cluster computing nodes have not completed theirinitialization and begun program execution within a predetermined periodof time), if the cost of using current computing nodes is sufficientlyhigh (e.g., for temporarily available computing nodes, such as if theircurrent associated price for use is above a defined threshold), etc.

Furthermore, in at least some embodiments, the dynamic modifying of theongoing distributed execution of a program on a cluster of multiplecomputing nodes may include actions other than dynamic scaling of thecluster. For example, the dynamic modifying of the distributed executionof a program may include replacing computing nodes in the cluster thatis executing the program with other computing nodes that have differentresource configurations and/or capabilities, may include replacingcomputing nodes in the cluster that is executing the program with othercomputing nodes from different sources, may include changing aparticular use of one or more computing nodes that are currently in acluster (e.g., switching a particular computing node between use as acore node and as an auxiliary node), etc. More generally, the dynamicmodifying of the ongoing distributed execution of a program on a clustermay include otherwise automatically adjusting resource usage by thedistributed program execution. For example, in at least someembodiments, the usage of one or more types of resources by one or morecluster computing nodes may be increased or decreased in variousmanners, such as by interacting with a module of the DPE service that islocal to those cluster computing nodes to temporarily or permanentlyexpand or restrict access to one or more types of computing resources bythe execution jobs of the program being executed on those computingnodes (e.g., to limit network bandwidth usage by restricting how manycommunications are allowed to be sent over the network on behalf of theprogram; to expand storage space available to the program by changing anamount of storage allocated to the program and/or by reducing the amountof storage on a shared storage device that is being used by one or moreother programs; etc.). In addition, in at least some embodiments,resource usage by one or more cluster computing nodes for distributedexecution of an indicated program may be modified in other manners, suchas by directly interacting with the indicated program to indicate achange in resource usage limits that are in use by the indicatedprogram, by reducing or expanding the amount of resource usage by one ormore other programs that share one or more types of computing resourceswith the indicated program so as to correspondingly expand or reduce theamount of resources available to the indicated program, by movingportions of the distributed program execution within the computing nodesof the cluster (e.g., to balance computing resource usage if somecluster computing nodes are under-utilized with respect to one or moretypes of computing resources and other cluster computing nodes areover-utilized with respect to those types of computing resources, bymoving computing-resource intensive portions of the distributed programexecution to cluster computing nodes that are configured or otherwisespecialized for that type of computing resource usage, etc.), by movingportions of the distributed program execution to different computingnodes outside the cluster (e.g., if the aggregate computing resourceusage for distributed execution of an indicated program is high withrespect to network bandwidth capacity but low with respect to othertypes of resource usage, changing the computing nodes of the cluster sothat the total quantity stays the same but the new cluster computingnodes are located closer together within the network or are providedwith access to network bandwidth that is not used by or isunder-utilized by other programs), etc. Such resource usage modificationmay be performed for various reasons, and may in some situations includetemporarily throttling usage of computing resources by the distributedprogram execution (e.g., if the distributed program execution is usingmore computing resources than allocated or otherwise expected, if one ormore bottlenecks exist with respect to executing programs of otherusers), terminating or temporarily suspending execution of the program(e.g., if an insufficient quantity of computing nodes of the cluster areavailable to perform execution), etc. Additional details related todynamically modifying the ongoing distributed execution of a program ona cluster of multiple computing nodes are included below.

As previously noted, the techniques may include managing the initialconfiguration to be used in the execution of an indicated program invarious manners, including using configuration parameters such as aquantity of computing nodes and/or other measures of computing resourcesto be used for the executing. The distributed execution of a program maybe initiated and configured in various manners in various embodiments,such as by a user interacting with an embodiment of a DPE service torequest the execution of the program in a manner specified by the user.For example, the DPE service may provide a GUI (“graphical userinterface”) that a remote user interactively uses to make the executionrequest (e.g., a Web-based GUI with which a user interacts via aclient-side application, such as a Web browser, executing on a clientdevice of the user), and/or the DPE service may provide one or more APIs(“application programming interfaces”) that enable a computing deviceand program of the user to programmatically interact with the DPEservice to make the request (e.g., one or more APIs that are also usedby the provided GUI). The user may specify various information as partof such a request, such as an indication of the program to execute(e.g., by supplying an executable copy of the program, by indicating aprogram previously registered by the user with the DPE service, byindicating a program made available by the DPE service for use by users,etc.), and an indication of input data for use by the executing program(e.g., by supplying the input data or by indicating an accessiblelocation from which the input data may be obtained). An executable copyof a program may be supplied in various manners and formats, such as,for example, a Java ARchive (“JAR”) file.

In addition, the user may further specify other configuration parametersfor the distributed program execution in at least some embodiments, suchas one or more of the following: an indication of a quantity ofcomputing nodes to use for the requested execution; an indication of oneor more resource configuration types of computing node to use for therequested execution (e.g., if the DPE service provides different typesof computing nodes with different capabilities, such as differentamounts of memory, storage and/or processing power); an indication ofone or more sources of computing node to use for the requested execution(e.g., if the DPE service provides computing nodes from differentsources with different prices and/or other use characteristics); definedscaling criteria for use in determining whether and how to dynamicallymodify the ongoing distributed execution of the program; one or moreother execution criteria to use in performing the requested execution(e.g., a user-specified QoS, or Quality of Service, level associatedwith the requested execution; an indication of a time by which therequested execution is to be completed; etc.); information about alocation or layout of some or all of the computing nodes to use for therequested execution, such as relative to each other, to indicated dataand/or to some other location (e.g., to be within a specified degree ofgeographical or network distance), or instead in an absolute manner(e.g., at a particular data center or other geographical location,within the boundaries of a country or other indicated geographical area,etc.); access information for use in communicating with one or more ofthe computing nodes, such as an SSH (“secure shell”) key; configurationinformation for use in configuring a particular instance of the programfor use with the requested execution; an indication of multipleexecution jobs into which the indicated program execution is to beseparated for parallel or other distributed execution, or instead anindication of how the DPE service is to automatically determine thosemultiple execution jobs (e.g., if the indicated program is designed tobe separable in a defined manner into distinct execution jobs, such asbased on an execution methodology used by or other design of theprogram); etc. More generally, in at least some embodiments, a user maybe able to specify other more general high-level execution criteria(e.g., to complete execution as cheaply as possible within someindicated time period, to complete execution as quickly as possible witha specified maximum associated fee, to complete execution in a mannerthat attempts to optimize one or more other types of indicated factors,etc.), and the DPE service may automatically determine preferred orotherwise appropriate execution configuration parameters to use tosatisfy those execution criteria, including by attempting to usetemporarily available computing nodes with sufficiently low associatedprices if available. Furthermore, in at least some embodiments, the DPEservice may further automatically determine some or all suchconfiguration parameters to otherwise use for the distributed executionof a particular program, such as for use in warning a user ifuser-specified configuration parameters are sub-optimal or otherwiseproblematic (e.g., are insufficient to complete the execution by adesired time indicated by the user), or for otherwise recommendingconfiguration parameters to the user. The automatic determining of suchconfiguration parameters may be performed in various manners in variousembodiments, such as based on the DPE service monitoring and assessingprevious related program executions for various users that use differingconfiguration parameters (e.g., execution of the same program, such asfor a program provided by the DPE service for use by users; execution ofother programs of the same or similar type, such as that use the samedesign or execution methodology; etc.) in order to identify one or morepreferred sets of configuration parameters for a particular program orparticular program type. Additional details related to configuring thedistributed execution of a program are included below.

The ongoing distributed execution of one or more programs for a user mayalso be tracked and modified in various manners in various embodiments,such as by a user interacting with an embodiment of a DPE service tomodify ongoing distributed program execution in a manner specified bythe user. For example, the DPE service may provide a GUI that a remoteuser may interactively use to view status information related to ongoingdistributed program execution (e.g., based on dynamic monitoring of theongoing distributed program execution that is performed by the DPEservice, based on status information gathered by a master node that iscontrolling the program execution or otherwise provided by the program,etc.) and/or to make a distributed program execution modificationrequest (whether the same GUI as used for configuring program executionor instead a distinct GUI), and/or may provide one or more APIs(“application programming interfaces”) that enable a computing deviceand program of the user to programmatically interact with the DPEservice to obtain such tracked information and make such modificationrequests (e.g., one or more APIs that are also used by the provided GUIfor use in modifying execution). Information that is provided to a usermay include, for example, one or more of the following: informationabout current use of different types of computing nodes within acluster, including information specific to core nodes and/or auxiliarynodes, information specific to computing nodes of one or more resourceconfiguration types, information specific to one or more computing nodesources, etc.; information about computing nodes that are available tobe used within a cluster, including information about computing nodes ofone or more resource configuration types and/or current price and/orother information about computing nodes available from one or moresources, including temporarily available computing nodes; etc. The typesof distributed program execution modifications that may be initiated bya user may vary in various embodiments and situations, including tomodify various previously specified configuration parameters for andistributed program execution (e.g., a quantity of computing nodes touse for the ongoing distributed execution), to suspend and optionallylater resume some or all of the ongoing distributed execution, etc.Additional details related to modifying the ongoing distributedexecution of a program are included below.

FIG. 1A is a network diagram that illustrates an example of a DPEservice that manages distributed execution of programs for multipleusers. For illustrative purposes, some examples and embodiments aredescribed below in which specific types of management of distributedprogram execution are provided in specific manners. These examples areprovided for illustrative purposes and are simplified for the sake ofbrevity, and it will be appreciated that the inventive techniques may beused in a wide variety of other situations, some of which are describedin greater detail below. For example, while particular examples of howto dynamically scale clusters used for the ongoing distributed executionof programs are described, other types of activities may be performed inother manners in other embodiments. In addition, while in someembodiments users initiate the execution of indicated programs in adistributed manner and/or modify the ongoing distributed execution ofindicated programs, in other embodiments an embodiment of the DPEservice may automatically initiate the execution and/or modification ofsuch indicated programs and/or may determine to execute a programindicated by a user in a distributed manner (e.g., without knowledge ofthe user). Furthermore, in some embodiments at least some of thedescribed techniques may be used with at least some programs that arenot executed in a distributed manner, such as to configure, monitorand/or modify execution of such a non-distributed program.

In the example of FIG. 1A, a number of users 140 are interacting over anetwork 100 with an illustrated embodiment of a Distributed ProgramExecution service that is implemented by one or more modules 110, suchas to initiate distributed execution of programs on one or morecomputing nodes 120 that are available for executing programs of theusers. The network 100 may have various forms, such as, for example, tobe a publicly accessible network of linked networks, possibly operatedby various distinct parties (e.g., the Internet). In other embodiments,the network 100 may be a private network, such as, for example, acorporate or university network that is wholly or partially inaccessibleto non-privileged users. In still other embodiments, the network 100 mayinclude one or more private networks with access to and/or from theInternet. In the illustrated embodiment, the DPE service modules 110 andthe computing nodes 120 are provided by a DPE service provider 105 aspart of an embodiment of a DPE service, as well as one or more optionalother storage nodes 130, and the DPE service modules 110 may execute onone or more other computing systems (not shown).

In some embodiments, the illustrated computing nodes 120 are provided bythe DPE service provider 105 for distributed execution of programs onbehalf of the users, and may include multiple physical computing systemsand/or multiple virtual machines that are hosted on one or more physicalcomputing systems (e.g., as is described in more detail with respect toFIG. 1B for one example embodiment). Each of the computing nodes 120 hassome amount of computing resources available for executing one or moreprograms, such as may be measured, for example, by a combination of oneor more of processing capacity (e.g., number and/or size of processingunits), memory capacity, storage capacity, disk I/O (“input/output”)capacity, etc., and other components (not shown) provided by the DPEservice may provide other computing resources (e.g., network bandwidthcapacity of one or more networks, additional storage capacity of networkstorage devices or other storage remote from particular computing nodes,etc.). In some embodiments, the DPE service provider 105 may providepreconfigured computing nodes, with each preconfigured computing nodehaving similar and/or equivalent amounts of computing resourcesavailable for executing programs on behalf of users, while in otherembodiments, the DPE service provider 105 may provide a selection ofvarious different computing nodes, such as with different types orgroups of computing nodes having varying amounts and/or types ofcomputing resources (e.g., size, speed and/or type of processing units;number of processing units; amount of memory and/or storage; platformconfiguration, such as 32-bit or 64-bit; etc.). In addition, thecomputing nodes 120 may include computing nodes that are available frommultiple sources and that have differing use characteristics. FIG. 2Aprovides additional details regarding an example embodiment in which thecomputing nodes 120 include computing nodes of multiple resourceconfiguration types and computing nodes from multiple different sources.

In the illustrated embodiment, the various users 140 may interact withthe DPE service modules 110 to make requests and specify variousinformation. Such user requests and specifications may be made atvarious times, such as when a user registers to use services of the DPEservice and/or at later times. For example, the DPE service modules 110may provide subscription and/or registration services to one or moreusers, such that a user may specify information related to one or moreprograms to execute on behalf of the user (e.g., programs, source code,network addressable locations of one or more programs, etc.), accountinformation (e.g., user name, billing information, etc.), terms of use,etc. In some embodiments, after a user interacts with the DPE servicemodules 110 to subscribe and/or register for services, the user may beissued one or more identifiers (e.g., keys, tokens, user names, etc.)that are associated with the user and are to be used in conjunction withexecuting programs on behalf of the user.

In addition, the users may interact with the DPE service modules 110 toinitiate and configure execution of programs in various ways in variousembodiments, such as by specifying a quantity and/or resourceconfiguration type of computing nodes for execution of programs, aminimum and/or maximum quantity of computing nodes to use, an initialconfiguration of core computing nodes and/or auxiliary computing nodesto use for a cluster, one or more defined scaling criteria for use indetermining whether to perform subsequent dynamic scaling of thequantity of computing nodes in a cluster or other modification of thecluster computing nodes, a preferred execution time and/or period ofexecution, an expiration time for the program execution request, aselection of one of multiple priorities for the execution (e.g., withassociated fees that increase with increased priority), informationabout one or more resource configuration types of computing nodes touse, information about one or more sources of computing nodes to use,etc. Furthermore, in some embodiments, a user may interact with the DPEservice modules 110 to request immediate execution of one or moreprograms on a specified number of computing nodes and/or to schedulesuch execution at one or more future times, such that the DPE servicemodules 110 may initiate the requested execution on the specified numberof computing nodes at a specified time or when one or more specifiedexecution criteria are satisfied (e.g., when it is determined that aspecified number of computing nodes are available).

In this illustrated embodiment, the DPE service provides a variety offunctionality for managing distributed execution of programs formultiple users on the computing nodes 120. For example, as previouslynoted, a particular user may use a GUI or API provided by the modules110 to submit a request for execution of an indicated program usingindicated input data, optionally along with a variety of other types ofconfiguration information. After the request for execution of theprogram is received, the DPE service modules 110 may select which of theavailable computing nodes 120 to use for the requested execution invarious ways. For example, in some embodiments, the modules 110 maysimply select an appropriate quantity of computing nodes from any of theavailable computing nodes with sufficient resources, such as, forexample, by randomly selecting from a single pool of available computingnodes. In other embodiments, one or more specific computing nodes may beselected on the basis of one or more other factors, such as, forexample, a predicted length of and/or likelihood of continuedavailability of the one or more computing nodes, a physical proximity ofthe one or more specific computing nodes to one or more other computingnodes, a geographic location of the one or more specific computing nodesand/or of one or more other computing nodes, a resource configurationtype of the computing nodes, one of multiple computing node pools orother sources of computing nodes, etc. In addition, after the request isreceived, the modules 110 may determine whether to use the multipleselected computing nodes for a cluster as core computing nodes and/orauxiliary computing nodes, and may further determine how to separate theindicated program into multiple execution jobs to be executed on some orall of the multiple selected computing nodes, such as by usinginformation supplied by the user and/or in an automatic manner based onpreviously obtained information about the design of the indicatedprogram. Similarly, in at least some embodiments and situations, themodules 110 may determine how to separate the indicated input data intomultiple subsets for use by the multiple execution jobs. For example, insome situations, each execution job may include executing a full copy ofthe indicated program but on a particular subset of input data, whilethe other execution jobs similarly execute the full program copyfunctionality on other input data subsets. Alternatively, in somesituations, various execution jobs may perform different functions on asingle common set of input data. The modules 110 may further determinewhich computing nodes of a cluster execute which execution jobs invarious manners, including in some situations based on the location inwhich input data to be used by an execution job is stored. As oneexample, an execution job may preferably be executed on a core computingnode that already stores some or all input data for the execution job insome embodiments. As another example, if the computing nodes of acluster are located in multiple distinct geographical and/or networklocations (e.g., in multiple distinct data centers), a particularexecution job may in some embodiments preferably be executed on one ofone or more computing nodes that are at a particular location thatstores at least one copy of the input data to be used for the executionjob.

As the execution jobs execute on the various computing nodes of acluster, the execution jobs store various information locally on thecomputing nodes in at least some embodiments, including as part of adistributed storage system on multiple core computing nodes, asdiscussed in greater detail elsewhere. In addition, the DPE service mayoptionally provide one or more storage nodes 130 that are used by theDPE service to store information related to program execution and/or forother purposes, such as to provide a reliable backup copy of data incase one or more computing nodes of the cluster become unavailable. Asdiscussed in greater detail elsewhere, such information stored on thestorage nodes 130 may include status information regarding theintermediate state of partial execution of various execution jobs forvarious programs, and in some embodiments may optionally include outputdata that is generated by completed execution jobs.

In addition, as the execution jobs of a program execute in a distributedmanner on the various computing nodes of a cluster for that program, theDPE service may automatically perform various actions to dynamicallymonitor and/or modify the ongoing distributed execution of the program,with FIGS. 2A-2E illustrating examples of types of dynamic monitoringand modifying that may occur in some situations. For example, asdiscussed in greater detail elsewhere, the dynamic modifying of theongoing distributed execution of the program on the multiple computingnodes of the cluster may include optionally performing various types ofchanges in certain situations, and the DPE service modules 110 mayselect which types of actions to pursue in which situations (e.g., basedon predefined scaling criteria specified generally for the DPE service,or defined scaling criteria specified specifically for the program beingexecuted or client on whose behalf the program is being executed). Forexample, if the DPE service modules 110 automatically determine todynamically add and/or remove computing nodes from the cluster, the DPEservice modules 110 may further select which computing nodes to add orremove, such as based on whether core computing nodes and/or auxiliarycomputing nodes are in use in the cluster, and/or in a similar manner tothe selections made initially by the modules 110 in selecting particularcomputing nodes for the cluster. In addition, if the DPE service modules110 automatically determine to make other types of changes to theongoing distributed program execution, the DPE service modules 110 maysimilarly determine which types of changes to make (e.g., how to reducebottlenecks corresponding to resource usage of the distributed programexecution by altering the distributed program execution in one or moreways, such as by altering which execution jobs and/or input data areused by particular computing nodes, throttling resource usage on some orall computing nodes of the cluster, stopping the distributed programexecution if sufficient cluster computing nodes are not available,etc.). Additional details related to the dynamic monitoring andmodifying of ongoing distributed program execution are describedelsewhere.

Furthermore, as discussed in greater detail elsewhere, in someembodiments the modules 110 may provide indications to cause theintermediate state of partial execution of one or more execution jobs tobe persistently stored before the partial execution of the execution jobis temporarily terminated or otherwise suspended. Such intermediatestate information for the partial execution may be stored in variousmanners remotely from the computing node on which the partial executionoccurred, such as by copying such intermediate state information to oneor more of the optional storage nodes 130 and/or by using one or moreoptional remote storage services 150 that are accessible over thenetwork 100. In some embodiments, the modules 110 coordinate the storageof the intermediate state information from the computing node to theremote persistent storage location, such as by using information thatthe modules 110 track about the intermediate state of the partialexecution, while in other embodiments the activities in performing thepersistent storage of the intermediate state information may instead beperformed by management software executing on the computing node tolocally manage the execution of the execution job. Alternatively, if aparticular executing execution job is able to store its own intermediateexecution state, the modules 110 may instead notify the execution job toperform its own intermediate execution state storage actions beforeshutting down its own execution. After the intermediate state of partialexecution of an execution job has been persistently stored and thepartial execution has been terminated, the partial execution may beresumed from that suspended intermediate state at a later time, such assubstantially immediately (e.g., if the execution of the execution jobis being moved from a first computing node to a second computing node,as quickly as the intermediate state information may be stored andretrieved), or instead after a longer period of time (e.g., at a laterscheduled time, a later time when one or more specified executioncriteria are satisfied, etc.). At the time of execution resumption, thestored intermediate state information may be retrieved from thepersistent storage location, and locally stored on or otherwise madeavailable to the computing node on which the execution job execution isto resume, and may include storing the data in a distributed storagesystem provided using multiple core computing nodes of the cluster. Inaddition, the partial execution of the execution job may be resumed invarious manners, such as by indicating to the execution job not toperform a subset of operations that were previously completed (e.g., ifthe execution job supports such functionality), by modifying theexecution job to only perform a subset of the operations that were notpreviously completed, etc. Additional details related to the suspensionand resumption of execution of execution jobs are described elsewhere.

FIG. 1B illustrates an embodiment in which a DPE service may be providedusing one or more data centers that include multiple physical computingsystems. In particular, FIG. 1B is a network diagram illustrating anexample embodiment in which one or more DPE service modules 180 of a DPEservice manage distributed execution of programs on behalf of users atan example data center 170. The example data center 170 is connected tothe Internet 196 external to the data center 170, which in this exampleprovides access to various external computing systems, such as computingsystems 190 via private network 194 and other directly accessiblecomputing systems 192. The private network 194 may be, for example, acorporate network that is wholly or partially inaccessible fromnon-privileged computing systems external to the private network 194.Computing systems 192 may include, for example, a home computing systemthat connects directly to the Internet (e.g., via a telephone or cablemodem, a Digital Subscriber Line (“DSL”), etc.). In addition, one ormore other data centers 198 are illustrated that are connected to datacenter 170 via the Internet 196 and may each include additionalcomputing systems, such as may further be used by the DPE service in atleast some embodiments. For example, the distributed execution of aparticular program may include simultaneous and/or sequential executionusing computing nodes at multiple data centers or other distinctgeographical locations, including to select a particular location atwhich to execute an execution job in various manners (e.g., based atleast in part on a location at which corresponding data is stored and/orat which one or more particular computing nodes of interest to be usedare located), and to optionally move execution of an execution job froma computing node at one geographical location to a computing node atanother geographical location.

The example data center 170 includes a number of physical host computingsystems 175, physical computing systems 182, optional storage systems160, and one or more DPE service modules 180 that provide an embodimentof the DPE service. In this example, host computing systems 175 eachprovide multiple virtual machines 177 and have a virtual machine (“VM”)manager module 179 to manage those virtual machines (e.g., a hypervisoror other virtual machine monitor), and some or all of the computingsystems 182 may similarly have one or more such virtual machines and/orVM manager modules (not shown). Such virtual machines may each act as acomputing node for use by the DPE service, and a VM manager module on ahost computing system provides one example of a type of module that theDPE service may use to dynamically monitor the host computing system(e.g., to obtain status information about one or more of the virtualmachine computing nodes hosted by that host computing system) and/or todynamically modify ongoing distributed program execution on one or moreof the virtual machine computing nodes hosted by that host computingsystem (e.g., by throttling computing resources used by one or more ofthe hosted virtual machine computing nodes, by changing computingresources allocated to or otherwise available to one or more of thehosted virtual machine computing nodes, etc.). Alternatively, in otherembodiments, some or all of the physical host computing systems at thedata center may not provide any virtual machines, such as to insteadeach act as a computing node that directly executes one or more programson behalf of end user customers of the DPE service, and such other hostcomputing systems may similarly execute a module of the DPE service thatmonitors and/or modifies distributed program execution in which thathost computing system participates. In addition, in some embodiments,the physical host computing systems 175 and/or physical computingsystems 182 may each include management software (e.g., a managementmodule of the DPE service, such as part of or distinct from the VMmanager modules) to manage the execution of execution jobs on thecomputing systems, whether as part of or instead distinct from the DPEservice module that performs the dynamic monitoring and/or modifying.Furthermore, in some embodiments various of the computing systems 175and 182 may have differing capabilities, may have different associatedfees for use, may support different types of user programs (e.g.,virtual machine software image instances of different sizes, or programswith different types of computing resource usage, such as differingpatterns of I/O and memory access and network usage), etc. If so, one ormore such factors may further be used as constraints and/or preferencesregarding which computing systems to select for executing particularprograms. The optional storage systems 160 may also have various forms,such as computing systems with attached storage (e.g., that may also beused as computing nodes at times), network storage devices that are notused as computing nodes, etc. The example data center 170 furtherincludes an internal network 172 that may include multiple networkingdevices (not shown), such as switches, edge routers, and core routers,with computing systems 175 and 182, storage systems 160, and the DPEservice modules 180 connected to the internal network 172.

The illustrated DPE service modules 180 perform at least some of thedescribed techniques in order to manage distributed execution ofprograms using the computing systems 175 and 182 (and optionally otherexternal computing systems), and to optionally persistently store atleast some program execution results on storage systems 160. Forexample, the DPE service modules 180 may provide a GUI or otherfunctionality that enables remote users to configure distributed programexecution and/or to track and optionally modify ongoing distributedprogram execution, such as for users (not shown) of computing systems190 or 192 or at one of the other data centers 198. When a particularcomputing node or system is selected to execute one or more executionjobs of a program, the DPE service modules 180 may in some embodimentsinitiate execution of the execution jobs by interacting with a VMmanager module or other manager module that controls execution ofprograms for that selected computing node/system, or may alternativelydirectly execute the execution jobs on the selected computingnode/system. The DPE service modules 180 may further in some embodimentstake various actions to dynamically monitor and/or modify the ongoingdistributed execution of various programs executing on the computingnodes of the data center 170 that are used by the DPE service. Inaddition, some of the computing systems 190 or 192 or at one of theother data centers 198 may be used to provide one or morenetwork-accessible remote storage services (not shown) that areaccessible by the DPE service and used to persistently store at leastsome intermediate results or other information.

It will be appreciated that the data center of FIG. 1B is provided forillustrative purposes only, and that an embodiment of a DPE service andother software execution services may be provided in other manners inother embodiments. For example, DPE service modules 180 may instead beprovided using one or more other computing systems external to the datacenter 170, such as computing systems 190, 192 or at a data center 198.In addition, in at least some embodiments, the programs for whichdistributed execution is provided may include executable softwareimages, such as virtual machine images that are bootable or otherwiseloadable on one or more virtual machine computing nodes, and that eachmay include operating system software, software for one or moreapplication programs, and/or configuration information, etc. At leastsome such programs may be stored by the DPE service and/or by users onthe storage systems 160 and/or using a remote storage service, and if soare retrieved by or supplied to the DPE service when they are to beexecuted on behalf of a user or as otherwise initiated. Furthermore,while only a limited number of devices are illustrated in this example,it will be appreciated that in a typical arrangement, data center 170may include hundreds or thousands of host computing systems such asthose illustrated here, organized into a large number of distinctphysical networks with a large number of networking devices (e.g., in ahierarchical manner).

As previously noted, FIGS. 2A-2E illustrate examples of types of dynamicmonitoring and modifying that some embodiments of a DPE service mayperform to manage the ongoing distributed execution of indicatedprograms.

In particular, FIG. 2A illustrates additional example informationregarding using the computing nodes 120 of FIG. 1A to perform thedistributed execution of one or more programs for one or more clients ofa DPE service embodiment. In this example, the computing nodes 120include at least one type 120 a of computing nodes, and optionallyinclude computing nodes of one or more additional distinct resourceconfiguration types 120 b-120 n. Each resource configuration type mayhave one or more computing nodes of that type, with each of thecomputing nodes of that type having a configuration of computingresources that differs from computing nodes of other resourceconfiguration types, as discussed in greater detail elsewhere. Inaddition, while not illustrated here, the available computing nodes 120may further include computing nodes that differ in one or more mannersother than their resource configuration, such as to have differentassociated costs, different conditions of use (e.g., a minimum period oftime and/or other specified conditions under which the computing nodewill continue to be available for use as part of the cluster, or theabsence of any such associated ongoing availability guarantees), etc.

Furthermore, the computing nodes 120 in this example are separated intomultiple different pools 215 of computing nodes that may each act as adistinct source of computing nodes for various client clusters, witheach pool having differing associated prices for using its computingnodes and/or having other differing use conditions related to using itscomputing nodes. The multiple pools 215 in this example include a pool215 b that may at various times include zero or more computing nodesthat are temporarily available for use in client clusters. Some or allof the computing nodes in the temporarily available pool 215 b may, forexample, represent excess capacity for computing nodes of a pool 215 athat may at various times include zero or more computing nodes that areallocated for one or more other dedicated uses. While such dedicated usecomputing nodes in pool 215 a may in some embodiments be used by the DPEservice in various manners (e.g., to allow clients to use their owndedicated use computing nodes within their own clusters, such as for anongoing fee that is less than a fee for using other on-demand computingnodes), the DPE service may in some embodiments and situations notdirectly use such computing nodes from pool 215 a in client clusters forother clients to whom the computing nodes are not allocated. In thisexample, the computing node sources further include a pool 215 c thatmay at various times include zero or more computing nodes that areallocated for on-demand variable capacity use by clients, as discussedin greater detail elsewhere.

In FIG. 2A, the computing nodes 120 are being used in multiple clusters205, including an example cluster 205 z that the DPE service uses toperform distributed execution of an example Program Z for example clientUser Z, as well as one or more other example clusters 205 x. FIGS. 2B-2Dprovide further example details regarding the distributed execution ofone or more such programs. In the example of FIG. 2A, the cluster 205 zincludes multiple core computing nodes 207 z, one or more auxiliarycomputing nodes 208 z, and one or more administrative computing nodes206 z, although other clusters may have other configurations, includingexample cluster 205 x that currently does not include any auxiliarycomputing nodes 208 z. The core computing nodes 207 z together provide adistributed storage system 209 z for use with cluster 205 z, with thedistributed storage system 209 z in this example storing various inputdata 235 to be used in the distributed execution of Program Z by cluster205 z and optionally storing various output data 245 that is generatedby the ongoing distributed execution of Program Z. While not illustratedhere, the auxiliary computing nodes 208 z and optionally zero or more ofthe core computing nodes 207 z may each be executing one or moreexecution jobs for Program Z. In addition, one or more of the corecomputing nodes 207, auxiliary computing nodes 208, and/oradministrative computing nodes 206 of each cluster may include nodes ofone or more resource configuration types 120 a-120 n. Similarly, one ormore of the core computing nodes 207, auxiliary computing nodes 208,and/or administrative computing nodes 206 of each cluster may includenodes from one or more pool sources 215 a-215 c.

In addition, the administrative computing nodes 206 z may include one ormore master nodes that manage the distributed execution of Program Zusing the core computing nodes 207 z and auxiliary computing nodes 208z, such as with at least one such administrative computing node actingas a name node to manage where copies of particular pieces or groups ofinput data are stored, and with at least one such administrativecomputing node acting as a job tracker node to manage the execution ofparticular execution jobs on particular computing nodes. For example, ajob tracker node may identify that a particular execution job isavailable to be executed, select a particular available core orauxiliary computing node to execute that execution job, and assign thatselected node to perform that execution job using indicated data. Thatselected node may then interact with the name node to identify alocation of the indicated data if needed (e.g., if the selected nodedoes not already store that indicated data), obtain the indicated datafrom the identified location if needed (e.g., if the selected node doesnot already store that indicated data), optionally obtain execution codefor the indicated execution job if needed (e.g., if the selected nodedoes not already store that execution code), and initiate execution ofthe indicated execution job using the indicated data.

As previously noted, the excess capacity computing nodes of pool 215 bmay represent excess and/or unused program execution capacity that maybe otherwise allocated for other uses. A request for using excesscapacity computing nodes of pool 215 b may be made in various ways invarious embodiments, such as by specifying one or more of the following:an exact number, a preferred number, a minimum number and/or a maximumnumber of computing nodes to use; an expiration time for the request; apreferred execution time and/or period of execution; one or moremonetary bids (e.g., a bid per each use of a computing node, a bid peruse of one or more computing nodes per some unit of time, a minimumand/or maximum bid, etc), etc.

In addition, as is further discussed with respect to FIG. 2E, theillustrated embodiment of the DPE service may determine when to includeand/or remove one or more computing nodes 120 from the pool 215 b. Invarious embodiments, the DPE service may determine that one or morecomputing nodes are unused and/or or otherwise available in variousways. For example, in some embodiments, the DPE service may receiveindications from various users or other clients that one or morecomputing nodes are not being used or are otherwise available for excesscapacity use, such as indications from one or more users of dedicatedcapacity pool 215 a computing nodes that they are not using some numberand/or portion of those computing nodes. In some such embodiments, suchdedicated capacity pool 215 a users may indicate one or more times atwhich their dedicated computing nodes are likely to be or are committedto be unused and/or available (e.g., particular times of day, particulardays, periods of time, etc.). In other embodiments, the DPE service mayautomatically determine when computing nodes are available for excesscapacity use, such as, for example, by monitoring some or all computingnodes 120 to determine when the computing nodes are not being usedand/or by tracking usage patterns of one or more users of such computingnodes. In some such cases, determining whether computing nodes areunused or otherwise underutilized may include determining and/orpredicting a likelihood that the computing nodes will remain unused forat least a period of time sufficient to execute programs of one or moreusers, such as, for example, based on an analysis of past use patternsof the computing nodes by one or more users. In various embodiments, aperiod of time sufficient to execute programs of one or more users maybe based on one or more considerations, such as a time to stop/startprograms on behalf of users, a time to configure computing nodes foruse, a type of program executed on behalf of users (e.g., some types ofprograms may perform useful amounts of work in short periods of time,such as various types of data processing, etc., while other programs uselonger periods of time), etc.

After it is determined that one or more computing nodes 120 areavailable for use in excess capacity pool 215 b, such computing nodesmay be tracked to determine if another use of the computing nodes arises(e.g., such as priority usage of one or more types). The DPE service maydetermine that one or more of the excess capacity computing nodes are nolonger available for use in pool 215 b in various ways in variousembodiments. For example, in some embodiments, the DPE service mayreceive indications that one or more computing nodes are no longeravailable, such as based on explicit requests to stop use of thecomputing nodes from a user to whom those computing nodes are allocated,based on requests to execute programs of priority users on the one ormore computing nodes, based on an expiration of a specified period ofavailability, etc. As another example, in some embodiments, the DPEservice may automatically determine other uses for the computing nodes,such as, for example, based on received requests from one or more usersthat correspond to the other uses, or based on determining a likelydemand for one or more computing nodes 120 (e.g., based on detecting anincreased usage of other resources and/or programs for which thecomputing nodes may be used, etc.).

In some embodiments, a client may interact with the DPE service torequest immediate use of a specified number of excess computing nodes inan indicated cluster and/or to schedule such use at one or more futuretimes, such that the DPE service may initiate the use of the specifiednumber of excess computing nodes if it is determined that the specifiednumber of excess computing nodes are available at the time of therequested execution. In addition, in some embodiments, a client mayinteract with the DPE service to configure a request for use of aspecified number of excess computing nodes in a cluster that is to beperformed in the future if the specified number of excess computingnodes becomes available, such as during an indicated future period oftime, and in some such embodiments the DPE service may initiate therequested use of the specified number of excess computing nodes when itdetermines that the specified number of excess computing nodes areavailable during that period of time. In some embodiments, a client mayspecify a minimum and/or maximum number of excess computing nodes to usefor an indicated cluster, such that the requested use is initiated ifthe DPE service determines that at least the minimum number of excesscomputing nodes are available, and the DPE service may initiate therequested use of up to the maximum (if specified) number of excesscomputing nodes based on availability of the excess computing nodes.

After a request for use of one or more excess capacity computing nodesis received, and if the DPE service determines that there are anappropriate number of computing nodes with sufficient computingresources to satisfy the request, the DPE service may select which ofthe available computing nodes in the pool 215 b to use for the requestin various ways. For example, in some embodiments, the DPE service maysimply select an appropriate number of excess computing nodes from anyof the available computing nodes with sufficient resources in the pool215 b, such as, for example, by randomly selecting from the pool. Inother embodiments, one or more specific computing nodes may be selectedfrom the pool 215 b on the basis of one or more other factors, such as,for example, a predicted length of and/or likelihood of continuedavailability of the one or more specific computing nodes, a physicalproximity of the one or more specific computing nodes to one or moreother computing nodes, a geographic location of the one or more specificcomputing nodes and/or one or more other computing nodes, etc.

As previously noted, use of excess computing nodes in a cluster may betemporary, such that the DPE service may automatically terminate suchuse when other preferred uses for the excess computing nodes arise(e.g., such as if the excess computing nodes are reclaimed by adedicated capacity user who previously was not using his/her dedicatedcomputing node, etc.). In such cases, the execution state of one or moreexecution jobs on such excess computing nodes may be saved before thetermination of use in some embodiments, such as to enable a laterrestart of those execution jobs. Furthermore, in some such embodiments,there may be multiple excess computing nodes of the pool 215 b that arecurrently in use in clusters that may be capable of satisfying thenumber of computing nodes for the other purposes, and in such cases, theDPE service may determine which of those excess computing nodes to freefor the other purposes based on various factors (e.g., by using adetermined priority among the current requests of the clients of theexcess capacity nodes, such as based on time submitted, bid prices,etc.). In some embodiments, at least some of the terminated use may bereinitiated on one or more other available excess computing nodes (ifany), such as immediately or at a later time (e.g., automatically asother excess computing nodes become available).

In at least some embodiments, access to excess computing nodes of pool215 b for use in client clusters may be based on priority among theclients, such that if there is contention for some amount of the excesscomputing nodes between two or more requests, the request having ahigher associated priority will be provided access to use the contendedamount of excess computing nodes. In at least one illustrativeembodiment, requests with the highest bid amounts (e.g., a highestmaximum bid) may be given priority over requests with lower bids, withties between bid amounts being resolved based on other factors (e.g.,which request was received earlier). In some embodiments, one or moreclients may interact with an embodiment of the DPE service to bid (e.g.,auction-style) on access to one or more available excess computing nodes(e.g., currently available and/or available at one or more futuretimes), such that the client with the winning bid may receive the accessto the one or more available excess computing nodes. In someembodiments, the DPE service may automatically terminate use of one ormore of the excess computing nodes associated with lower priorityrequests in favor of higher priority requests for the one or more excesscomputing nodes. In some other embodiments, such as non-priority basedembodiments, the DPE service may instead occasionally rotate throughpending requests to use excess computing nodes, such that each requestmay be provided some amount of use of one or more of the excesscomputing nodes.

In some embodiments, the DPE service may provide interactive feedback toone or more of the client that are interacting with the DPE service toconfigure and/or request use of some amount of excess computing nodes.For example, in some such embodiments, interactive feedback may includeindications of when and/or for how long use of the indicated amount ofexcess computing nodes is likely to last, such as based on currentand/or predicted demand, current and/or likely amounts of excesscomputing nodes, one or more bid-levels, etc. In one illustrativeembodiment, the DPE service may indicate suggested bid levels along withcorresponding information indicating when requested use will likelyoccur and/or complete, such that the client may reconfigure (e.g., byspecifying a different bid, a different amount and/or type of computingnodes, a different duration of execution, etc.) a request to meet thedesires of the client, such that the request may be performed at anearlier time, etc.

As previously noted, one or more of the client may be charged variousfees in conjunction with use of excess capacity computing nodes, such asbased on computing nodes used and/or one or more use factors (e.g.,number of times used, amount of shared resources consumed, amount oftime computing nodes are used, etc.), and/or based on one or more bidsfrom the one or more clients for use of the computing nodes. In someembodiments, a portion of the fees charged to the one or more clientsmay be supplied to one or more other users who provided computing nodesfor use by the clients (e.g., one or more users of dedicated capacitycomputing nodes in pool 215 a). For example, in some embodiments, suchother users may be paid a proportional share of some amount of the feesthat are collected from clients, such as a proportional share reflectiveof the amount of computing nodes contributed by the other users overtime. In some cases, such fees supplied to the other users may beautomatically credited and/or paid to the other users by the DPEservice, such as to offset other charges incurred by those other users,such as charges incurred by for use of dedicated capacity nodes in pool215 a. In addition, in at least some additional embodiments, unusedportions of a single one of the computing nodes in pool 215 b (e.g.,unused processing unit clock cycles, unused portions of memory, etc.)may be made available for use by one or more clients, such that acluster of a first client may be sharing resources of a single computingnode with a user to whom a dedicated capacity computing node isallocated and/or with other client clusters.

In addition, although some embodiments are described with respect to aDPE service and execution capacity of computing nodes, it will beappreciated that the described techniques may be used to manage accessto various types of computing-related resources. A non-exclusive list ofexamples of other types of computing-related resources that may bemanaged for use by multiple users may include the following: persistentdata storage capabilities (e.g., on non-volatile memory devices, such ashard disk drives); temporary data storage capabilities (e.g., onvolatile memory, such as RAM); message queuing and/or passingcapabilities; other types of communication capabilities (e.g., networksockets, virtual communication circuits, etc.); database managementcapabilities; dedicated bandwidth or other network-related resources;input device capabilities; output device capabilities; CPU cycles orother instruction execution capabilities; etc.

FIG. 2B illustrates various example status information 210 thatcorresponds to the ongoing distributed execution of an example programat a particular time, and in particular of Program Z on cluster 205 z ofFIG. 2A at Time 1. In the example of FIG. 2B, various example computingnodes of cluster 205 z are participating in the distributed execution ofProgram Z, as discussed in greater detail with respect to FIG. 2C, andone or more other computing nodes (not shown) may also be part of thecluster and acting as master computing nodes to control various actionsof other cluster computing nodes. The status information 210 may reflectstatus information that is gathered by automatic monitoring by anembodiment of the DPE service (not shown) and/or by one or more of themaster computing nodes. The status information 210 may be stored andused in various manners, and in some embodiments may be used by the DPEservice in automatically modifying the distributed execution of ProgramZ.

In the example of FIG. 2B, the status information 210 is displayed aspart of a GUI screen 285 that also includes various user-selectablecontrols 220. Such a GUI may be displayed to, for example, a humanoperator user of the DPE service, and/or User Z of the DPE service whoinitiated the distributed execution of Program Z. In this example, theuser-selectable controls include a “Modify” control 220 a via which theuser may supply instructions to modify the ongoing distributed executionof Program Z in one or more manners as specified by the user (e.g., toadd or remove computing nodes from the cluster, to modify computingresource usage of Program Z, etc.). In addition, in this example, theuser may use the “Suspend” control 220 b to temporarily suspend theongoing in-progress distributed execution of Program Z, the “Terminate”control 220 c to permanently terminate the ongoing in-progress executionof Program Z, and the “Duplicate” control 220 d to initiate execution ofa new program by duplicating at least some of the specifiedconfiguration information of a selected program via control 220 b. Insome embodiments, other controls may be available, such as a “Resume”control (not shown) to resume the execution of a previously suspendedprogram, whether immediately or at an indicated future time, or moredetailed controls to control the dynamic scaling of the cluster (e.g.,separate controls to increase or decrease the quantity of computingnodes in the cluster; controls that affect the use of core and auxiliarynodes, such as to select a particular type of node whose quantity willbe modified or to more generally indicate whether a particular scalingoperation is intended to be permanent or temporary; etc.). It will beappreciated that other types of controls may be displayed in otherembodiments, and that controls may be displayed and used in variousmanners in various embodiments.

In this example, the status information 210 includes various executionstate information regarding the distributed execution of Program Z, suchas to track the status of execution of execution jobs on the multiplecomputing nodes used for the distributed execution. In particular, inthis example, the example time of Time 1 reflects a time that is shortlyafter distributed execution of Program Z has been initiated (e.g., 10minutes), and thus the distributed execution of Program Z is not yetfully using the cluster computing nodes. In particular, computing nodeNode D 207 z-d has not yet begun executing any execution jobs forProgram Z, such as if Node D is still being initialized for Program Z(e.g., is still obtaining input data to be used, is still obtainingsoftware code corresponding to one or more execution jobs of Program Zto be executed on Node D, is still configuring the Program Z softwarecode before execution begins, is still establishing access to the masternode and/or to other cluster computing nodes, etc.), or has encounteredone or more problems (e.g., a failure or other unavailability, abottleneck caused by another executing program, etc.). Computing nodeNode E 208 z-e is similarly not yet executing any execution jobs forProgram Z, but has just completed its initialization process, and isready to begin executing its allotment of Program Z execution jobs. Theother 3 computing nodes 207 z and 208 z that are included in the initialcluster for Program Z have already begun their respective portions ofthe distributed execution of Program Z at Time 1, with those other nodesbeing Node A 208 z-a, Node B 207 z-b, and Node C 208 z-c. In thisexample, nodes A, C and E are auxiliary computing nodes 208 z and nodesB and D are core computing nodes 207 z, although the distributed storagesystem 209 z is not illustrated, and the local storage by each node ofits corresponding input data 235 and output data 245 is shown.

In addition, in this example, each line or entry in the information 210corresponds to the performance of a particular operation for aparticular execution job on a particular computing node, withinformation being tracked that in this example includes anidentification 210 a of the computing node, of the execution job 210 b,of the operation 210 c, of the status of performance 210 f of theoperation, of input data 210 d to be used by the operation, of outputdata 210 e to be produced by the performance of the operation, andoptionally of various other information 210 g. Such other statusinformation may include, for example, information about dependencies orother inter-relationships between operations (e.g., operation B cannotbe executed until after operation A is completed, operations C and D areto be executed simultaneously, etc.), information about expectedcompletion of performance of an operation being performed (e.g., anexpected completion time, an expected amount of time until completion, apercentage completion of an operation, a percentage of the operationthat remains to be performed, etc.), information about expectedinitiation of performance of operations that are not yet beingperformed, etc. Various other types of information may similarly bedisplayed for Program Z, such as information about a current stage orphase of the execution, about the one or more master nodes that controlexecution of the execution jobs on the various cluster computing nodesand that may optionally gather and store various status information forthe distributed execution, a graphical depiction of some or all of thevarious multiple computing nodes, etc.

While not illustrated in this example, in some embodiments andsituations, different users or other entities may be involved indifferent parts of the distributed execution of a program, such as tohave a first user initiate distributed execution of a program, a seconduser initiate a first suspension of execution of that program, a thirduser initiate a first resumed execution of that program, etc. Such twoor more different users may, for example, represent a single client, notrepresent the same client but be affiliated in one or more other manners(e.g., based on working together), have authorization that has beengranted to perform specified actions for other users, etc.

In the example of FIG. 2B, the execution of an example execution job J-Ahas been initiated on computing node Node A. Execution job J-A includesoperations to be performed in this example that include at least exampleoperations A01, A02, A03, and A04, and the first four entries in thestatus information 210 correspond to these operations for this executionjob and computing node. In this example, operations A01 and A02 havebegun and are in progress, with operation A01 using input data D-A01 andwith operation A02 using input data D-A02. The graphical indications ofcluster computing nodes 207 z and 208 z are provided in FIG. 2B forillustrative purposes to indicate information about the current statusof the computing nodes, but are not displayed as part of the GUI screen285 in this example. In particular, the graphical display of Node A 208z-a indicates the input data D-A01 235 a and D-A02 235 b that arecurrently loaded on Node A and being used, as well as indications 245 aand 245 b of output data for operations A01 and A02 that are in theprocess of being produced. In particular, operation A01 will produceoutput data D-A11 245 a, which operation A03 uses as input data—sincethat data D-A11 245 a is not yet available, operation A03 is queued tobe executed while Node A is waiting for that data to be available (andoptionally for sufficient computing resources to be available on Node Ato execute the operation). Similarly, operation A02 is in progress andis using input data D-A02 to eventually produce output data D-A12, andsince operation A04 uses that output data D-A12 as input data, operationA04 is similarly queued and ready to be executed when its input data andNode A are available to perform it. Similar status information isillustrated for other computing nodes, execution jobs and operations.

FIG. 2C continues the example of FIG. 2B, and in particular illustratesthe changing status of the distributed execution of Program Z over time.The left portion of FIG. 2C corresponds to Time 1, and thus thegraphical depictions of the computing nodes 207 z and 208 z match thoseshown in FIG. 2B at the same time. As discussed in greater detailelsewhere, in some embodiments the distributed execution of Program Zmay be stopped at Time 1 or a prior time, such as if the 3 examplecomputing nodes that have begun executing portions of Program Z justprior to Time 1 (those being Node A, Node B and Node C) are notdetermined to be a sufficient quantity or percentage of the clustercomputing nodes (e.g., less than 90% participating in the distributedexecution of Program Z within 10 minutes of the Time 0 initializationtime for the Program Z distributed execution).

In the illustrated example, however, the distributed execution ofProgram Z is not stopped (or is stopped but is later resumed orrestarted prior to Time 2), and the status of the distributed executionof Program Z at Time 2 is illustrated in the center of FIG. 2C. In thisexample, the progression of the distributed execution on the variouscluster computing nodes has progressed. For example, with respect toNode A, operation A01 has now been completed, as reflected by the outputdata D-A11 240 a as being shown as now being stored on Node A, andfurther being used as input for the now in-progress performance ofoperation A03 that is in the process of generating additional outputdata D-A21 245 k. Since input data D-A01 has now been used, it mayoptionally have been removed from Node A, and operation A02 iscontinuing to be performed and in the process of generating output dataD-A12 245 b. The other computing nodes have similarly progressed intheir distributed execution of Program Z, including that Node D and NodeE are both now participating in the distributed execution of Program Z.

However, some modifications have been automatically made to the ongoingdistributed execution of Program Z since Time 1 in this example. Inparticular, the cluster has been dynamically scaled to increase thetotal quantity of cluster computing nodes by at least 1, with newauxiliary computing node Node F 208 z-f being illustrated and in use atTime 2 as part of the distributed execution of Program Z. The dynamicscaling may have been initiated in various manners, such as by anexplicit instruction received from User Z or by the satisfaction of oneor more defined scaling criteria for the cluster. In addition, the nodeadded may have been of any of the resource configuration types 120 a-120n that are available and/or from any of the pools 215 a-215 c or othersources, such as is indicated by the instruction or defined scalingcriteria, or is instead automatically selected by the DPE service. Ifone or more defined scaling criteria were used to initiate the dynamicscaling, the defined scaling criteria may, for example, have beensatisfied based on the execution status of one or more particular nodesor based on the aggregate execution status of the cluster. In thisexample, Node C may have experienced slower performance than expected(e.g., based on Node C having less available computing resources ascompared to other cluster computing nodes; based on a bottleneck beingcreated on a host physical computing system, not shown, that providesNode C, such as by another computing node provided by that host physicalcomputing system as part of the distributed execution of anotherprogram; etc.), and/or the overall distributed execution of Program Zmay be taking longer than expected (e.g., based solely on the unexpecteddelay with respect to Node C, based on unexpected slowness of one ormore cluster computing nodes other than Node C; etc.). If so, the DPEservice may have proceeded to automatically add Node F to the cluster,and in this example has distributed a portion of the Program Zoperations that were previously assigned to Node C to now be performedby Node F. In particular, Node C continues to perform the operation C01,as was previously occurring at Time 1, but operation C02 has beentransferred to Node F and is currently being performed by Node F at Time2 (as demonstrated by the input data D-C02 235 f being loaded on Node Fand the output data D-C12 245 k being in the process of being generatedon Node F by operation C02). Node F also in this example is executingone or more other operations that were not previously assigned to any ofthe cluster computing nodes at Time 1, such as operation F01 (not shown)that uses input data D-F01 235 k and is in the process of generatingoutput data D-F11 245 f.

FIG. 2C further illustrates the status of the distributed execution ofProgram Z at a later Time 3, which is illustrated on the right side ofFIG. 2C. In the illustrated example, much of the distributed executionof Program Z has been completed, and dynamic scaling has been used toreduce the quantity of computing nodes in the cluster by Time 3. As withthe dynamic scaling that occurred prior to Time 2, the later dynamicscaling may have been initiated in various manners, such as by anexplicit instruction received from User Z or by the satisfaction of oneor more defined scaling criteria for the cluster. In addition, as withthe adding of Node F, the nodes to be removed may have been of any ofthe resource configuration types 120 a-120 n that are available and/orfrom any of the pools 215 a-215 c or other sources, such as is indicatedby the instruction or defined scaling criteria, or is insteadautomatically selected by the DPE service (e.g., to remove a node frompool 215 b because it is planned to be put to a different higherpriority use). In this example, Node A and Node C have completed theportions of the distributed execution assigned to them (both the initialassignments, and any later assignments) before Time 3, and are alsoauxiliary computing nodes 208 z that are preferably selected for dynamicremoval in lieu of any core computing nodes 207 z. Accordingly, whileother cluster computing nodes continue the ongoing distributed executionof Program Z, the DPE service has automatically selected Node A and NodeC to be removed from the cluster. Thus, in this manner, the DPE servicemay automatically modify the ongoing distributed execution of Program Zand the corresponding cluster used for that distributed programexecution at various times and for various reasons. Clusters may bedynamically scaled in various other manners and for various otherreasons, as discussed elsewhere.

FIG. 2D continues the examples of FIGS. 2B and 2C, and in particularillustrates additional status information regarding the ongoingdistributed execution of Program Z, as well as other programs by the DPEservice. In this example, status information 290 and 291 is shown thatindicates status at Time 2, such as may be automatically gathered byautomatic monitoring activities of the DPE service. In addition, thestatus information 290 and 291, as well as other status information 292and 293, may be part of a GUI screen 295 that is displayed to one ormore users in a manner similar to that previously discussed with respectto FIG. 2B, although no user-selectable controls are illustrated in theexample of FIG. 2D, and/or the various status information may be usedinternally by the DPE service as part of automatically determiningwhether to perform dynamic scaling for the distributed execution ofProgram Z, such as based on the satisfaction of one or more definedscaling criteria. While not illustrated in FIG. 2D, in some embodimentsthe status information that is displayed may further include a varietyof types of information about different types of computing nodes used ina cluster or that are otherwise available (e.g., about core computingnodes versus auxiliary computing nodes of a cluster; about computingnodes of different resource configuration types in a cluster, optionallyin a manner that is also specific to core computing nodes and auxiliarycomputing nodes; about computing nodes in a cluster that are availablefrom different sources, such as with different associated prices and/oruse conditions, optionally in a manner that is also specific to corecomputing nodes and auxiliary computing nodes; about computing nodesthat are available to be added to a cluster, such as computing nodes ofdifferent resource configuration types and/or that are available fromdifferent sources; etc.), including to display such information aboutongoing execution of a program on a cluster to a client associated withthe cluster in order to enable the client to dynamically modify theongoing execution of the program on the cluster.

In the example of FIG. 2D, various status information 290 has beenmonitored regarding the distributed execution of Program Z at Time 2,including various aggregate computing resource usage information. Inparticular, in this example, each line or entry in the information 290corresponds to a particular one of the cluster computing nodes, withinformation being tracked that in this example includes anidentification 290 a of the computing node, of one or more correspondingexecution jobs 290 b being executed on that computing node, of thestatus of performance 290 c of the execution job(s), and variouscomputing resource usage information 290 d-290 h. The example computingresource usage information that is shown in this example includes aquantity 290 d of disk I/O for an indicated hard disk that the computingnode is using (e.g., an average over a prior period of time, a currentpoint-in-time value, etc.) and a percentage 290 e of the total disk I/Ofor that indicated hard disk that the computing node is using, aquantity 290 f of network bandwidth I/O for an indicated local networkthat the computing node is using (e.g., an average over a prior periodof time, a current point-in-time value, etc.) and a percentage 290 g ofthe total network bandwidth capacity for that indicated network that thecomputing node is using, etc. Various other types of computing resourceusage information 290 h may similarly be shown, and it will beappreciated that the various types of computing resource usageinformation may be displayed in various manners in various embodiments.

In addition to the computing resource usage data for the distributedexecution of Program Z that is shown for each of the cluster computingnodes, the bottom of the status information 290 also shows aggregatecomputing resource usage information for the entire distributedexecution of Program Z. For example, the illustrated data includes atotal quantity 290 d of disk I/O used by all of the cluster computingnodes, an average percentage 290 e of the total disk I/O used by all ofthe cluster computing nodes, a total quantity 290 f of network bandwidthI/O used by all of the cluster computing nodes, and an averagepercentage 290 g of the total network bandwidth capacity used by all ofthe cluster computing nodes. In this example, most of the hard disks andnetworks being used by the cluster computing nodes are shared computingresources used by other computing nodes (e.g., other computing nodes ofthe same cluster, other computing nodes of other clusters that areexecuting other programs, etc.), although example hard disk N used byNode E is not shared (“NSh”) by others (e.g., is a remote logicalnetwork disk that is used only by Node E), and thus in this example theusage of that non-shared computing resource is not included in theaggregate computing resource usage information. In other embodiments,all computing resources may be shared or may be not shared, and theaggregate information may or may not include both types of computingresource usage information.

In addition to the status information 290 illustrated for the ongoingdistributed execution of Program Z, FIG. 2D also illustrates otherstatus information 291 corresponding to the ongoing distributedexecution of other programs of the DPE service, although only aggregatecomputing resource usage information is shown for these other exampleprograms. Furthermore, FIG. 2D also illustrates various statusinformation 292 from monitoring the total usage of networks of the DPEservice by the various programs being executed, and may similarlyinclude status information 293 from monitoring the total usage of harddisks by the various programs of the DPE service being executed, as wellas optionally other types of computing resource usage.

The various status information of FIG. 2D may be used in variousmanners, including by the DPE service as part of automaticallydetermining whether to modify ongoing distributed execution of one ormore programs of the DPE service, such as in accordance with one or moredefined scaling criteria for particular programs and their clusters. Forexample, with respect to the ongoing distributed execution of Program Z,the usage of disk J (by Node A, at 70% of the total disk I/O) andaggregate usage of disk L (by Nodes C and F, at an aggregate 95% of thetotal disk I/O) may exceed an allocation or other expected usage forshared disks, and thus may create a bottleneck for any other programsthat are attempting to use those disks. As such, the DPE service maydetermine to take various actions, such as to throttle the usage ofthose disks by those computing nodes (e.g., of the usage by one or bothof Nodes C and F of disk L), or to take other actions to accommodate theactual or potential bottleneck (e.g., temporarily prevent any othercomputing nodes from using disk L, so that the aggregate 95% total diskI/O that is being used by Program Z does not create any actualbottlenecks for other programs). In other embodiments and situations,the DPE service may not take action until an actual bottleneckarises—for example, if other computing nodes are not attempting to use30% or more of the total disk I/O for disk J, then Node A's use of 70%of the total disk I/O may be allowed until additional I/O capacity ofdisk J is requested by other computing nodes. Similarly, with respect tonetwork bandwidth usage, the aggregate use of network T by Nodes D and Eis 80% of the total network bandwidth capacity, and another 10% is beingused by one or more other programs (as shown in the I/O capacity 292 a,I/O usage 292 b and I/O % 292 c of status information 292). Accordingly,such network T usage by Nodes D and E may be throttled due to thepotential or actual bottleneck caused by the use of network T for thedistributed execution of Program Z. As another alternative, one or bothof Nodes D and E may be automatically determined by the DPE service tobe replaced or supplemented by one or more other computing nodes to beadded to the cluster for Program Z that are part of other local networksof the DPE service, such as based on network T having relatively lowtotal bandwidth capacity compared to other networks of the DPE service(e.g., network S with ten times more bandwidth capacity and a currenttotal bandwidth usage of 24%).

FIG. 2E illustrates an example of techniques for managing use of excesscomputing nodes that are temporarily available for use in clusters whennot otherwise being used, such as that may be automatically performed byan embodiment of the DPE service. In particular, FIG. 2E includes atime-based table 200 that illustrates information about usage of anumber of computing nodes 210 by various users over time 205, with useof each computing node 210 being illustrated at consecutive timeintervals 205 a-205 o. In addition, table 250 illustrates informationabout example requests A, B, C and D received to use computing nodes onbehalf of users. In the illustrated table 200, a time interval having asolid gray background (e.g., time interval 205 a of dedicated computingnode 212 b, and all time intervals of other computing node 214 a)indicates that the respective computing node is not available to be usedfor excess program execution capacity during that respective interval oftime (e.g., it is being used for one or more other purposes and/or isotherwise not available); a time interval having a solid whitebackground (e.g., time intervals 205 b and 205 c of dedicated computingnode 212 c) or having a diagonal-lined background (e.g., time interval205 f of dedicated computing node 212 b and time interval 205 o of othercomputing node 214 b) indicates that the respective computing node isavailable for use as excess program execution capacity during thatrespective interval of time; and a time interval containing a letter A,B, C or D (e.g., time intervals 205 f-205 k of dedicated computing node212 a) indicates that programs of the respective request for excessprogram execution capacity are executing on the respective computingnode during that time interval.

As previously noted, table 250 illustrates various requests received atvarious indicated times, such that each of the various requests is touse a specified amount of excess computing nodes as part of a clientcluster. In addition, each of the requests has various specifiedconfiguration information in this example, including a specified minimumand maximum number of computing nodes for the request; a bid fee amountthat the user is willing to pay for use of each of the computing nodes(e.g., such as a dollar amount per use hour of a single computing node);and an indication of a time at which the request is to expire (e.g.,such as a specified total duration of execution time after which arequest is to expire, a specified time at which a request is to expire,etc.). As one illustrative example, request A is received at a time thatapproximately corresponds to time interval 205 c (e.g., just before timeinterval 205 c or during time interval 205 c), with a minimum andmaximum number of computing nodes both indicating a preference toexecute on a single excess capacity computing node, with a bid price of$0.05 per hour of use of the single excess computing node, and with nospecified expiration (e.g., indicating that the request is tocontinuously be considered until it is fully satisfied). In otherembodiments, the various requests may be configured in other ways, suchas to include one or more of a specified particular type or source ofcomputing node to use, a minimum and/or maximum bid amount, and/or oneor more other configurations (e.g., fault tolerance requirements,execution locality and/or proximity preferences, etc.).

The illustrated table 200 includes a number of dedicated capacitycomputing nodes 215 a, such as computing nodes that have been allocatedfor dedicated access to one or more specific users. For example, adedicated capacity user (not shown) may have priority access to thespecific computing node 212 a for a specified period of time (e.g., ayear), such that the user may access computing node 212 a on behalf ofthe user at any time during the specified period of time, although suchaccess may be subject to a delay period and/or one or more interactionson the part of the user to gain access (e.g., notifications of an intentto use the computing node 212 a at a particular time, a request to usecomputing node 212 a as part of a particular cluster, etc.). In otherembodiments, the dedicated capacity user (not shown) may instead havepriority access to a computing node with equivalent computing resourcesas computing node 212 a (e.g., equivalent processing capacity, memory,etc.), but not have a particular allocated computing node 215 a, suchthat the user may be provided access to any of the computing nodes 215 athat are equivalent to 212 a for executing programs on behalf of theuser and that are available for use. In various such embodiments, theDPE service may ensure that a sufficient number of dedicated computingnodes equivalent to 212 a are available for use by dedicated users whomay have priority access to such computing nodes in various ways (e.g.,maintaining a pool of such computing nodes and/or otherwise reserving aspecific amount of such nodes, etc.).

In addition, the illustrated table 200 indicates a number of othercomputing nodes 215 b, such as computing nodes that may be available forother types of program execution capacity (e.g., on-demand variableprogram execution capacity). In this illustrated embodiment, all thecomputing nodes 210 are treated as being equivalent (e.g., havingequivalent computing resources), although other embodiments may existwhere various of the computing nodes 210 may be of different types, suchthat various of the requests may include indications of one or morespecific types of the computing nodes to use for executing programs insatisfaction of the requests.

In table 200, during interval 205 a-c, dedicated computing node 212 chas been determined to be excess capacity, such as based on being unusedby a dedicated capacity user to whom computing node 212 c is allocated,and has thus been made available for excess capacity use (e.g., beenadded to a pool of temporarily available excess capacity computingnodes). For example, the dedicated capacity user may have interactedwith the DPE service to indicate that computing node 212 c is availablefor excess program execution capacity, such as at some time prior totime interval 205 a, or the DPE service may have instead automaticallydetermined that the computing node 212 c was unused. In the illustratedexample, all other of the illustrated computing nodes (e.g., 212 a-b,212 d, and 214 a-b) have not been determined to be available during theinterval 205 a-205 c. During time interval 205 c, request A is received,and the DPE service determines to satisfy request A using the availableexcess capacity computing node 212 c, for the user-specified fee of$0.05/hour, and initiates execution of the program on computing node 212c, such as by adding computing node 212 c to a client cluster (notshown) associated with request A (e.g., as an auxiliary node). In otherembodiments, the DPE service may instead have a fixed price fee for someor all excess capacity use.

At time interval 205 d, the DPE service determines that computing node212 c is no longer available to satisfy excess program executioncapacity requests (e.g., based on an indication received from adedicated capacity user reclaiming use of the computing node), and sothe use of the excess capacity node associated with request A isterminated for computing node 212 b (e.g., computing node 212 b isremoved from the cluster associated with request A). At interval 205 f,the DPE service determines that two computing nodes with sufficientresources to satisfy request A (e.g., dedicated computing nodes 212 aand 212 b) are available as excess capacity computing nodes, anddetermines to continue satisfaction of request A by using dedicatedcomputing node 212 a as an excess capacity node added to the respectiveclient cluster. In the illustrated embodiment, the diagonal lines attime interval 205 f with respect to dedicated computing node 212 bindicate that node 212 b is available during that time interval for useas excess computing capacity, but is not preferred for excess computingcapacity use, and thus is not selected to satisfy request A. Computingnode 212 b may not be preferred for various reasons, such as it having ashort and/or uncertain duration of availability (e.g., as determined bythe DPE service, such as based on indications from the dedicatedcapacity user to whom it is allocated, based on prior history of use,based on forecasted use, etc.), whereas node 212 a may be determined tohave an availability that is likely to be relatively longer than node212 b. In such a situation, the DPE service may have a preference forselecting a computing node with a longer likely availability forsatisfying a request for an excess capacity computing node, such as tominimize having to start/stop use of the various computing nodes 210within clusters, etc. Thus, in this example, computing node 212 b is notused during time interval 205 f, but if another excess computingcapacity request had been received prior to time interval 205 f, node212 b may have nonetheless been used for that other request.

In this example, request B is received at interval 205 g, as illustratedby table 250. In this case, there is only one excess computing nodeavailable at that time (e.g., dedicated computing node 212 a), and theDPE service determines to terminate the satisfaction of request A inlieu of request B, such as based on request B having a higher bid amount(e.g., $0.08/hour) than request A (e.g., $0.05/hour) or otherwise havinga higher priority. In this example, request B is satisfied continuouslyusing dedicated computing node 212 a for a fee of $0.08/hour over thetime intervals 205 g-205 i. At time interval 205 i, two computing nodesare available as excess computing nodes (dedicated computing nodes 212 aand 212 d), each having sufficient resources for satisfying requests Aand B, and in the illustrated embodiment request B continues to use node212 a while satisfaction of request A is reinitiated on node 212 d.

At time interval 205 j, three dedicated computing nodes are determinedto be available as excess capacity computing nodes (computing nodes 212a, 212 c and 212 d), and request C is received. In this embodiment, thesatisfaction of requests A and B using dedicated computing nodes 212 dand 212 a, respectively, is terminated, and all three of the availableexcess computing nodes are selected for use in satisfying request Cbased on request C having an indicated preference to execute on amaximum of three nodes and having a relatively higher bid amount (e.g.,$0.10/hour) than requests A and B. At time interval 205 k, two dedicatedcomputing nodes are determined to be available as excess computing nodes(computing nodes 212 a and 212 c), such as based on computing node 212 dhaving been reclaimed for other purposes or otherwise having becomeunavailable. The use of computing node 212 d for request C is terminatedbut the use of nodes 212 a and 212 c for request C continues, such asbased on the preferred maximum and minimum number of computing nodes forrequest C, and request C having a higher priority than requests A and B.After time interval 205 k, the DPE service determines that request C hasbeen satisfied for at least two hours over the time intervals 205 j-205k, and has thus expired. Accordingly, the DPE service terminates use ofthe excess capacity computing nodes for request C, and request C is nolonger treated as a pending request to be satisfied. At time interval205 l, one computing node (dedicated computing node 212 c) is determinedto be available for use as excess program execution capacity, and thesatisfaction of request B is reinitiated on the computing node.

At time interval 205 m, an indication is received (not shown) to cancelrequest B (e.g., an indication from the client who submitted request B),such that the use of computing node 212 c with request B is terminated,and request B is treated as no longer being a pending request to besatisfied. In addition, at or near this time, request D is received,having a bid amount equivalent to previously received request A (e.g.,$0.05/hour). In this case, the DPE service determines to reinitiatesatisfaction of request A on available dedicated computing node 212 crather than request D, based on request A having been received at anearlier time than D. Request A continues to use node 212 c until someunknown future time after time interval 205 o. At interval 205 o, one ofthe other computing nodes 214 b becomes available for use as excessprogram execution capacity, and is determined to be a non-preferredexcess capacity computing node, but the program of request D isnonetheless executed on the other computing node 214 b since no otherexcess computing nodes are currently available.

In the illustrative example of FIG. 2E, the DPE service may furthertrack usage of the computing nodes 210 corresponding to the variousrequests A-D, such that the various associated client are charged feescorresponding to the bid amounts indicated by the various requests. Inaddition, the DPE service may also track which of the computing nodes210 were used and/or were made available for use by excess capacityusers, such that one or more users associated with those computing nodesmay be given some portion of the fees collected from the excess capacityusers.

It will be appreciated that the preceding example is provided forillustrative purposes, and that other embodiments may differ in variousways from the example. For example, although the computing nodes 210 aredepicted as being equivalent (e.g., having equivalent computingresources), other embodiments may be provided where various of thecomputing nodes 210 may be of different types with varyingcharacteristics (e.g., different amounts of processing capacity, memory,platform specification, etc.). In some such embodiments, various of therequests may include indications of one or more specific types of thecomputing nodes for use in executing programs associated with therequests, and those requests may only be executed on the correspondingspecified type of computing node. In addition, in some embodiments,rather than excess computing capacity being based on unused dedicatedcomputing nodes and other computing nodes as illustrated, embodimentsmay exist where only one group of computing nodes and/or additionalgroups of computing nodes may contribute to excess capacity computingnodes available for executing requests of excess capacity users.Furthermore, in some embodiments, at least some of the other computingnodes 215 b may include computing nodes provided to the DPE service byone or more third parties.

It will be appreciated that the illustrated and other types of statusinformation may be automatically monitored by the DPE service in variousmanners, and may be automatically used by the DPE service in determinemodifications to make to ongoing distributed execution of programs. Inaddition, it will be appreciated that the information in the examples ofFIGS. 2A-2E has been provided for illustrative purposes only, andvarious details have been shown in an abstract manner or not illustratedfor the sake of understanding. Furthermore, it will be appreciated thatwhile only a limited number of computing nodes and execution jobs havebeen illustrated, in actual embodiments the actual quantities may bemuch larger, such as to include hundreds or thousands or millions ofcomputing nodes and/or execution jobs.

As previously noted, the DPE service may provide various functionalityto perform distributed execution of programs for users. For example,after program execution is requested or otherwise initiated, whether bya user or in another manner, that program execution may be separatedinto multiple execution jobs for parallel or other distributed executionin various manners in various embodiments. As previously noted, theprogram may be designed to be separable in a defined manner intodistinct execution jobs (e.g., to support task-level parallelism), andif so a user may indicate how to generate multiple execution jobs and/orthe DPE service may automatically generate the multiple execution jobs.Furthermore, some types of programs may perform various operations oneach of a number of pieces of input data, and in such situations, dataparallelism techniques may be used to separate the group of input datafor use in the program execution into multiple subsets that are eachsupplied to a distinct execution job (e.g., with each execution jobperforming some or all of the same operations on the input data subsetused for that execution job). As one simple example, a particularprogram execution may include searching a large number of Web pages fortwo particular terms (or alternatively indexing the Web pages for use inlater searches), and the program execution may be separated into a largenumber of execution jobs that execute simultaneously in parallel to eachsearch (or index) a distinct subset of the Web pages (e.g., a single Webpage, or a small number of the Web pages) for one or both of the terms.Non-exclusive examples of types of programs that may be separated intomultiple execution jobs (e.g., to support task parallelism and/or dataparallelism) include the MapReduce program for processing and generatinglarge data sets by performing at least a first phase of various mapfunctions and a second phase of various reduce functions, the Hadoopprogram implementation of MapReduce execution methodology, the Hive datawarehouse infrastructure that uses Hadoop, the Pig structured querylanguage for use with HDFS (“Hadoop Distributed File System”) data, theHBase open-source distributed database, various types of image, videoand speech processing, various types of analysis of and searchingthrough textual data sets or other types of input data, etc. Additionaldetails related to the MapReduce program are included in “MapReduce:Simplified Data Processing on Large Clusters” by Jeffrey Dean and SanjayGhemawat, December 2004, Proceedings of OSDI'04: Sixth Symposium onOperating System Design and Implementation, which is incorporated hereinby reference in its entirety, and which is also available at the time ofapplication filing atlabs<dot>google<dot>com<slash>papers<slash>mapreduce-osdi04.pdf (where“<dot>” and “<slash>” are replaced with corresponding characters “.” and“/”).

In addition, in at least some embodiments, the dynamic monitoring of theongoing distributed execution of a program may include monitoring someor all execution jobs for the program, such as to determine whenparticular execution jobs begin execution and complete execution. Suchmonitoring of the execution of execution jobs may provide variousbenefits, such as to determine when to later initiate execution of otherexecution jobs (e.g., if the output data generated by a first executionjob is at least some of the input data for a second execution job, so asto assist in determining when to execute the second execution job),and/or to determine when all execution jobs of a program have beencompleted. Furthermore, in at least some embodiments, some or allexecution jobs may each have multiple distinct operations (which alsomay be referred to as “tasks” in some situations) that are to beperformed, such as in a sequential and/or parallel manner, and themonitoring or other tracking of the ongoing execution of such executionjobs may further include determining information about the status of thepartial performance of some or all such operations, such as to reflectintermediate data that is generated by completed performance of somesuch operations. Such tracking of the performance of particularoperations for an execution job may further provide various benefits,such as to enable the ongoing intermediate execution and data state fromthe partial execution of the execution job to be tracked and used, asdescribed in greater detail below. The monitoring may include using asystem manager module to initiate execution of execution jobs onparticular computing nodes, and to subsequently obtain statusinformation from the computing nodes (e.g., by the system manager modulepulling status information from the computing nodes, such as byperiodically requesting status information from each computing node,and/or by the computing nodes pushing status information to the systemmanager module, such as periodically or when particular events occur).

When executing an execution job for a program on a computing node,various information related to the execution may be stored in variousmanners. For example, when the execution is initiated, the input data tobe used by the execution job may be locally stored on the computing node(e.g., temporarily on a local hard disk or other local storage device)to facilitate access to that input data during execution, and anysoftware instructions to be executed for the execution job may similarlybe locally stored on the computing node, such as if with one or bothtypes of data being retrieved from a distributed storage system for acluster that is executing the program. As one example, if a distributedprogram is being executed using an implementation of MapReduce executionmethodology, the input data and/or output data used by a computing nodemay be stored on a local portion of a distributed HDFS storage system onthat computing node. After the execution of the execution job of theprogram is completed, the local storage on the computing node may insome embodiments be erased or otherwise cleared after any output datafrom the execution is copied back to the distributed storage systemand/or to a remote long-term storage location used by the DPE service,such as in preparation for or as part of initiating execution of anotherexecution job on the computing node (e.g., another execution job of adifferent program for a different user). In other embodiments, the localstorage on the computing node may instead be maintained even after theexecution of an execution job is completed (e.g., until execution of theprogram is complete). Furthermore, in at least some embodiments, variousintermediate output data and other intermediate results from the partialexecution of an execution job on a computing node may be temporarilystored locally on the computing node during the execution, such as tocorrespond to output data produced by a subset of the operations of theexecution job whose performance is completed, and/or to correspond topartial or incomplete output data produced by one or more operationswhose performance has been initiated but not completed. In at least someembodiments in which a long-term storage location of the DPE service isused, such intermediate results are not stored on the DPE service'slong-term storage location (e.g., to minimize execution delay andnetwork traffic involved in copying such intermediate results to the DPEservice's long-term storage location), while in other embodiments someor all such intermediate results may be stored on the DPE service'slong-term storage location.

As one example of intermediate results, referring back to the priorsimplified example of an execution job that involves searching throughmultiple Web pages for two terms, each operation of the execution jobmay correspond to searching through a single Web page for a single term,with the intermediate output data from performance of such an operationbeing information about any occurrences of that term in that Web page(e.g., a cumulative number of times that the term is located;information about each occurrence of the term in that Web page, such asits location in the Web page; etc.). Alternatively, as another example,a particular execution job may involve searching through a Web log orother transaction record that has numerous entries (e.g., with each linein the Web log representing an entry) for a particular term, and eachoperation of the execution job may involve searching a particular inputdata entry for that term, with the intermediate output data fromperformance of such an operation similarly including information aboutwhether the input data entry includes the term.

By storing the output data from the completed execution of an executionjob in the DPE service's long-term storage location, the availability ofthat output data may be maintained even if one or more particularcomputing nodes subsequently fail or otherwise become unavailable, suchas if those computing nodes are core computing nodes whose failureaffects the use of the distributed storage system in which theyparticipate. However, if intermediate results from partial execution ofan execution job are not stored on the DPE service's long-term storagelocation, a termination of the execution of that execution job beforecompletion may result in those intermediate results being lost, suchthat the execution job would need to later be executed again from thebeginning (e.g., to repeat performance of all operations of theexecution job, even if some of the operations were previouslycompleted). Accordingly, distributed execution of a program is managedby the DPE service in at least some embodiments in such a manner as tostore and use such intermediate results from partial execution of anexecution job that is temporarily terminated or otherwise suspendedbefore completion, so that a later resumption of the execution of theexecution job may resume at or near the intermediate point of partialexecution before termination. For example, if a determination is made toterminate execution of one or more execution jobs of a program on one ormore computing nodes before the execution of those execution jobs iscomplete, a module of the DPE service may initiate the persistentstorage of the intermediate results from the partial execution of thoseexecution jobs, such as at a location remote from those computing nodes.When the execution of those execution jobs is later resumed, whether onthose same computing nodes or other computing nodes, the persistentlystored intermediate results from the prior partial executions may beretrieved and used in the resumed execution in various ways. Forexample, if multiple operations of a particular terminated execution jobhad been completed before the termination, the intermediate results fromthe performance of those operations may be retrieved for the resumedexecution, and those completed operations need not be performed againfor the resumed execution.

The termination of partially completed execution jobs may be performedfor various reasons in various embodiments. For example, in someembodiments a user who has requested execution of a program may requestthat the execution of some or all execution jobs for the program besuspended, such as for an indicated period of time or until the userlater requests a resumption of the program suspension. In otherembodiments, the DPE service may initiate the termination of theexecution of one or more execution jobs of a program. For example, theexecution of an execution job on a first computing node may beterminated and moved to another second computing node, such as if thefirst computing node is to be shut down for maintenance, is to be usedfor another execution job or other program (e.g., another execution jobor other program with a higher priority), is being over-utilized, isshowing signs of possible failure, is over-using one or more types ofcomputing resources, etc. In addition, in a manner similar to that ofthe indicated user request, the DPE service may determine to suspend allexecution of a program for a period of time.

In addition, as previously noted, in at least some embodiments,execution of an execution job that is automatically terminated may beautomatically resumed at a later time by the DPE service. For example,in some such cases, the DPE service may continue to try to execute suchan execution job until completion of the requested execution, untilexecution is cancelled or otherwise explicitly terminated by a user,until occurrence of a specified time (e.g., until occurrence of anexpiration time, until occurrence of a cumulative execution time, etc),until occurrence of a specified number of executions, indefinitely, etc.In addition, in at least some such embodiments, at least some executionjobs that are automatically terminated may be automatically resumed onbehalf of the user at a future time when a sufficient amount of programexecution capacity again becomes available to continue the execution.Similarly, if the ongoing distributed execution of a program is stoppeddue to an insufficient quantity of computing nodes of a cluster beingavailable to perform the distributed execution or otherwise havinginsufficient computing resources to perform the distributed execution,the DPE service may reschedule the distributed execution for a latertime and/or automatically restart the distributed execution at a futuretime when a sufficient quantity of computing nodes or other sufficientamount of program execution capacity again becomes available to continuethe execution.

As previously noted, various information may be locally stored on acomputing node during execution of an execution job, and the locallystored information may be stored in various manners. For example, insome embodiments, a distributed file system or other distributed datastore may be created on multiple core computing nodes of a cluster usedfor distributed execution of a program, with a particular core computingnode's locally stored data being stored in the local portion of thatdistributed data store. Furthermore, if the distributed data store for aparticular embodiment includes redundant data that allows for recoveryof the data locally stored on a particular computing node that fails orotherwise becomes unavailable, then the availability of the output datafrom the execution of an execution job may be maintained in manysituations for that embodiment even without the use of separatelong-term storage for the DPE service. In other embodiments, some or allinformation may be stored in other manners, such as by not using along-term storage location of the DPE service (e.g., by maintaininglocally stored data on computing nodes until all execution for theprogram is completed) and/or by not using local storage on the computingnodes selected to assist in the distributed execution of a program(e.g., by storing any information that is not loaded in volatile memoryon the computing node at a remote location, such as the DPE service'slong-term storage location).

In addition, when intermediate results of partial execution on acomputing node are persistently stored remotely from the computing node,the intermediate results may be stored in various manners. In someembodiments in which the DPE service uses a long-term storage location,the intermediate results may be stored in the DPE service's long-termstorage location along with other completed results of execution jobsthat have been completed. Alternatively, in some embodiments, theintermediate results for a particular computing node may be storedtogether, but in a manner distinct from the intermediate and/orcompleted results of other computing nodes. For example, in someembodiments, a remote logical storage volume may be created to mirrorsome or all of a local hard disk or other block data storage device forthe computing node that was used to store the intermediate results,including a local portion of a distributed file system or otherdistributed data store. When the partial execution of the execution jobfor the computing node is later resumed, such a remote logical storagevolume may be attached to the computing node on which the resumedexecution is to occur, such as to allow the information in the attachedlogical storage volume to be copied to a physical block storage deviceof that computing node for use during the resumed execution, or theattached logical storage volume may instead be used during the resumedexecution in lieu of any physical block storage device of that computingnode. In some such embodiments in which logical storage volumes areavailable for use, the logical storage volumes may be created at thetime of termination of an execution job on a computing node, while inother embodiments the logical storage volume for an execution job may becreated at the beginning of the initial execution of the execution jobon a computing node and used throughout the entire execution (e.g., inlieu of any physical block storage device of that computing node), evenif the execution of the execution job is terminated and resumed one ormore times.

Additional details related to the operation of examples of remotestorage services that may be used to persistently store program stateinformation and/or other information, including logical storage volumes,are available in U.S. patent application Ser. No. 12/188,943, filed Aug.8, 2008 and entitled “Providing Executing Programs With Reliable AccessTo Non-Local Block Data Storage;” and in U.S. patent application Ser.No. 11/371,304, filed Mar. 8, 2006 and entitled “Distributed StorageSystem With Web Services Client Interface” and claiming priority benefitof U.S. Provisional Patent Application No. 60/754,726 filed Dec. 29,2005, each of which is hereby incorporated by reference in its entirety.In addition, additional details related to example embodiments ofexecuting different programs of different priorities at different timesand to techniques for suspending and resuming distributed execution ofprograms are included in U.S. patent application Ser. No. 12/334,022,filed Dec. 12, 2008 and entitled “Managing Use Of Program ExecutionCapacity,” and in U.S. patent application Ser. No. 12/334,004, filedDec. 12, 2008 and entitled “Saving Program Execution State,” each ofwhich is hereby incorporated by reference in its entirety. Furthermore,additional details related to enabling users to configure and modifydistributed program execution are included in U.S. patent applicationSer. No. 12/415,649, filed Mar. 31, 2009 and entitled “ManagingDistributed Execution Of Programs;” in U.S. patent application Ser. No.12/415,725, filed Mar. 31, 2009 and entitled “Dynamically Modifying ACluster Of Computing Nodes Used For Distributed Execution Of A Program;”and in U.S. patent application Ser. No. 12/415,773, filed Mar. 31, 2009and entitled “Dynamically Monitoring And Modifying Distributed ExecutionOf Programs;” each of which is also hereby incorporated by reference inits entirety.

In addition, as previously discussed, the managing of distributedexecution of programs may be performed in various manners in variousembodiments. For example, the determination of which computing nodes touse for execution of a program may be made in a variety of ways,including based on any preferences and/or requirements specified inconfiguration information that accompanies initiation of programexecution or otherwise specified for the program and/or associated user(e.g., at a time of registration, etc.). For example, if preferredand/or required resources for execution of a program are identified(e.g., memory and/or storage; CPU type, cycles or other performancemetric; network capacity; platform type, etc.), the selection ofappropriate computing nodes to use may be based at least in part onwhether the computing nodes have sufficient resources available tosatisfy those identified resources. In at least some situations, theinitiation of the distributed execution of the indicated program on acluster by the DPE service includes some or all of the followingnon-exclusive actions: selecting multiple computing nodes to be used asthe cluster to perform the distributed execution of the indicatedprogram; identifying one or more of the multiple nodes of the cluster toact as master nodes that control the execution of the program on theother nodes of the cluster; provisioning the multiple computing nodes ofthe cluster if needed to prepare them to receive software to be executedand input data to be used; obtaining the indicated input data to be usedby the distributed execution of the indicated program; separating theobtained input data into multiple subsets that are each to be used onone of the multiple computing nodes of the cluster; separating theindicated program into multiple execution jobs to be executed on themultiple computing nodes of the cluster; for each of the multiplecomputing nodes, loading on the computing node software and optionallyinput data to be used for the distributed execution of the indicatedprogram (e.g., software to execute at least one execution job for thecomputing node and a determined subset of the indicated input data to beused by the computing node); optionally configuring the loaded softwareon the various computing nodes (e.g., configuring the software loaded onthe one or more master computing nodes to be able to use the othercomputing nodes of the cluster); establishing access controls for themultiple computing nodes of the cluster that enable the multiplecomputing nodes to inter-communicate as part of the executing of theindicated program; etc. Additional details related to executing andconfiguring programs to execute on a program execution service areincluded in pending U.S. patent application Ser. No. 11/395,463, filedMar. 31, 2006 and entitled “Managing Execution of Programs by MultipleComputing Systems,” which is hereby incorporated by reference in itsentirety.

In some embodiments, fees may be associated with the use of a DPEservice, such that the DPE service may perform distributed execution ofprograms on behalf of a user in exchange for payment of one or more feesby that user. For example, in some embodiments, fees may be charged to auser based on an amount and/or type of distributed program executioncapacity allocated for executing one or more programs on behalf of auser, such as based on one or more of the following non-exclusive list:a number of computing nodes in a cluster, a number of processing units,an amount of memory, an amount of storage, an amount of networkresources, etc. In some embodiments, fees may be based on other factors,such as various characteristics of the computing resources used toexecute programs, such as, for example, based on CPU capabilities orperformance, platform type (e.g., 32-bit, 64-bit, etc.), etc. Fees mayalso be charged on the basis of a variety of use factors in someembodiments, such as a price per use of the service, a price per unit oftime that computing services are used, a price per storage used, a priceper data transferred in and/or out, etc. In at least some embodiments, aprovider of a DPE service may offer one or more of various tiers, typesand/or levels of services or functionality for distributed execution ofprograms on behalf of multiple users, and in some such embodiments,various fees may be associated with the various tiers, types and/orlevels of services. Additional details related to various feesassociated with a distributed program execution service are included inpending U.S. patent application Ser. No. 11/963,331, filed Dec. 21, 2007and entitled “Providing Configurable Pricing for Execution of SoftwareImages,” which is hereby incorporated by reference in its entirety.

In addition, as previously noted, in at least some embodiments the DPEservice provides an API to enable users to programmatically configurethe distributed execution of programs and to modify ongoing distributedprogram execution. The following include non-exclusive examples offunctions that may be provided as part of such an API: describe-cluster;run-cluster; terminate-cluster; set-cluster-proxy; upload-mapped-jar;upload-data; start-program; cancel-program; and get-program-status, inwhich “jar” refers to a file containing the indicated program to beexecuted. Furthermore, in at least some embodiments users may be enabledto configure a variety of characteristics for their clusters, including,for example, a degree of replication regarding how many copies (if any)to maintain of intermediate data during program execution, memoryallocation to be used for particular programs (e.g., for use with a Javavirtual machine), a loglevel for which information is tracked duringdebugging, etc. It will be appreciated that an API may include otherfunctionality and be defined in other manners in other embodiments, andthat the various functions of the API may each have appropriatevariables or parameters whose values are specified as part of invokingthe function.

In addition, in at least some embodiments, the DPE service may provideother types of functionality in at least some situations. For example, auser may initiate the distributed execution of a first program on acluster of multiple computing nodes, but may maintain the cluster ofmultiple computing nodes even after the distributed execution of thefirst program has ended. One reason that the user may maintain thecluster is to execute a distinct second program on the existing clusterafter the first program has ended, such as a second program that usesthe same or similar configuration (e.g., the same type of program butwith a new input data set), or instead a second program that usesgenerated results or other output data from the execution of the firstprogram as input data for the distributed execution of the secondprogram. As another example, in some embodiments, a user may be allowedto specify input data for use in the distributed execution of a programthat is not static, such as if the input data continues to expand orotherwise change while the program is executing—one example is a logfile for a Web site or data that indicates ongoing transactions, withthe distributed execution of the program analyzing each entry in the logor transaction data, such that the new log/transaction entries areanalyzed as they are received (or later if the processing of thelog/transaction data has not yet reached those new entries at the timeof their receipt), and another example is a program performingtransaction processing services for ongoing transactions. Furthermore,in some embodiments a user may specify one or more types of limitsregarding the distributed execution of a program (e.g., an amount ofexecution time; a cost of execution; an amount of usage of one or moretypes of computing resources, such as memory, storage, disk I/O, networkI/O; etc.), with various specified types of actions that the DPE serviceis to take if a specified limit is reached (e.g., to notify the user, tosuspend or terminate execution of the program, to reduce usage of a typeof resource corresponding to the limit, etc.).

Furthermore, various other types of functionality may be provided andused by a DPE service in various embodiments, as discussed in greaterdetail elsewhere.

FIG. 3 is a block diagram illustrating an example embodiment of a systemsuitable for performing techniques to manage distributed execution ofprograms. In particular, FIG. 3 illustrates a server computing system300 suitable for executing an embodiment of a Distributed ProgramExecution system 340, as well as various user computing systems 350,computing nodes 360, and other computing systems 380. In the illustratedembodiment, the server computing system 300 has components that includea CPU 305, various I/O components 310, storage 320, and memory 330. Theillustrated I/O components include a display 311, a network connection312, a computer-readable media drive 313, and other I/O devices 315(e.g., a keyboard, a mouse, speakers, etc.). In addition, theillustrated user computing systems 350 have components similar to thoseof server computing system 300, including a CPU 351, I/O components 352(although particular components are not illustrated), storage 354, andmemory 357. The other computing systems 380 and computing nodes 360 mayalso each include similar components to some or all of the componentsillustrated with respect to server computing system 300, but suchcomponents are not illustrated in this example for the sake of brevity.In addition, the computing nodes 360 may be of multiple types in atleast some embodiments, including of multiple resource configurationtypes, and/or may be available from multiple sources, including at leastone source of temporarily available computing nodes, as discussed ingreater detail elsewhere.

An embodiment of the Distributed Program Execution system 340 isexecuting in memory 330, such as under control of CPU 305 as programmedwith software instructions of the system 340, and it interacts withcomputing systems 350 and 380 and computing nodes 360 over the network390 (e.g., via the Internet and/or the World Wide Web, via a privatecellular network, etc.). In this example embodiment, the DPE system 340includes functionality related to managing distributed execution ofprograms on computing nodes 360 by various users (not shown) who areinteracting with user computing systems 350, such as in conjunction witha network-accessible DPE service provided by the DPE system 340. Theother computing systems 350 and 380 and computing nodes 360 may beexecuting various software as part of interactions with the DPE system.For example, user computing systems 350 may each be executing a Webbrowser 358 or other software in memory 357, such as to interact with aUser Interaction Manager module 342 of the DPE system 340 in order toconfigure and/or request execution of programs on behalf of the users ofthose systems on one or more computing nodes 360 in various ways, totrack and optionally request modifications to ongoing distributedexecution of programs, and/or to perform various other types of actions.

Various information related to the operation of the DPE system 340 maybe stored in storage 320, such as information 322 related to thecomputing nodes used to execute particular programs, information 328about clients of the distributed program execution service, information324 that includes information about program execution (e.g., finalexecution results from program execution that has been completed, statusinformation about the intermediate state of partial execution of variousprograms, etc.), user-specified and other configuration information 326used to configure distributed execution of programs (e.g., specifiedexecution configuration parameters, defined scaling criteria related todynamic cluster modification, etc.), and information 329 regarding thestatus of resource usage by executing programs and indications ofdetected bottlenecks. As discussed elsewhere, at least some of theillustrated information may be stored in other manners, such as if someor all of the program execution information 324 is stored in adistributed storage system (not shown) provided using various of thecomputing nodes 360, whether instead of or in addition to storing thatinformation on storage 320. In addition, various intermediate stateinformation and other information may be persistently stored in variousmanners, as discussed in greater detail elsewhere, including on adistributed storage system provided using various of the computing nodes360 and/or on storage 320 of server computing system 300. In addition,in some embodiments other storage systems may be used to providelong-term storage of information for some or all clusters, as describedelsewhere, including by using local or remote other computing systems380 and/or other computing nodes/systems or storage nodes/systems (notshown).

After the DPE system 340 receives requests (or other indications) toexecute one or more programs on one or more computing nodes 360, alongwith specified configuration information, the DPE system 340 selects thecomputing nodes for the cluster that will perform the distributedexecution of the one or more programs, determines whether to use theselected computing nodes as core nodes or auxiliary nodes for thecluster, and initiates execution of execution jobs of those programs onat least some of those computing nodes 360 of the cluster. In addition,the DPE system 340 may further interact with computing nodes 360 totemporarily terminate or otherwise suspend execution of execution jobson the computing nodes and to resume previously terminated execution,such as in response to user instructions. The DPE system 340 may alsodynamically monitor or otherwise interact with one or more of thecomputing nodes 360 to track use of those computing nodes, such as undercontrol of one of the optional other modules 349 of DPE system 340, andmay further dynamically modify the ongoing distributed execution ofprograms on the computing nodes 360, such as to dynamically scale thecluster of computing nodes under control of the Cluster ModificationManager module 347 of DPE system 340. In some embodiments, the modules347 and 349 may interact in various manners, such as if the module 347requests or otherwise obtains information from one or more of themodules 349, while in other embodiments one of the modules 349 may storemonitored information as part of information 329 and/or 324, and themodule 347 may retrieve and use such stored information as desired.Furthermore, in some embodiments the DPE system 340 may optionallyinclude a Program Configuration Manager module 344 that uses informationgathered from the monitoring of the distributed execution of variousprograms to facilitate configuring the distributed execution of otherprograms, such as by determining preferred execution configurationparameters for particular predetermined types of programs and/or bydetermining recommended execution configuration parameters forparticular indicated programs, although in other embodiments the module344 may not be provided. If present, the Program Configuration Managermodule 344 may further interact with the User Interaction Manager module342 in some situations, such as to provide determined configurationinformation to the module 344 for presentation to users, while in otherembodiments such determined configuration information may be stored bymodule 344 as part of configuration information 326, and retrieved andused by module 342 from the stored configured information 326.Additional details related to these operations are included elsewhere.

The computing nodes 360 may have various forms in various embodiments,such as to include a number of physical computing systems and/or anumber of virtual machines executing on one or more physical computingsystems. In some embodiments, the server computing system 300 andcomputing nodes 360 may be part of a data center or other group ofco-located computing systems, or may otherwise be computing nodes of aprivate network. In addition, in some embodiments, the DPE system 340may interact with one or more other computing systems 380 to initiate,suspend or resume execution of one or more programs on those computingsystems, such as if the computing systems 380 are provided by one ormore third-party participants. As described elsewhere, the computingnodes 360 may be of multiple types in at least some embodiments,including of multiple resource configuration types and/or in multipledistinct locations, and may in some embodiments be available fromvarious sources that have differing associated fees for use and/orassociated use conditions, including a pool of temporarily availablecomputing nodes.

It will be appreciated that computing systems 300, 350 and 380 andcomputing nodes 360 are merely illustrative and are not intended tolimit the scope of the present invention. The computing systems and/ornodes may instead each include multiple interacting computing systems ordevices, and the computing systems/nodes may be connected to otherdevices that are not illustrated, including through one or more networkssuch as the Internet, via the Web, or via private networks (e.g., mobilecommunication networks, etc.). More generally, a computing node or othercomputing system may comprise any combination of hardware or softwarethat may interact and perform the described types of functionality,including without limitation desktop or other computers, databaseservers, network storage devices and other network devices, PDAs, cellphones, wireless phones, pagers, electronic organizers, Internetappliances, television-based systems (e.g., using set-top boxes and/orpersonal/digital video recorders), and various other consumer productsthat include appropriate communication capabilities. In addition, thefunctionality provided by the illustrated DPE system 340 may in someembodiments be distributed in additional modules. Similarly, in someembodiments some of the functionality of the DPE system 340 may not beprovided and/or other additional functionality may be available.

It will also be appreciated that, while various items are illustrated asbeing stored in memory or on storage while being used, these items orportions of them may be transferred between memory and other storagedevices for purposes of memory management and data integrity.Alternatively, in other embodiments some or all of the software modulesand/or systems may execute in memory on another device and communicatewith the illustrated computing systems via inter-computer communication.Furthermore, in some embodiments, some or all of the systems and/ormodules may be implemented or provided in other manners, such as byconsisting of one or more means that are implemented at least partiallyin firmware and/or hardware (e.g., rather than as a means implemented inwhole or in part by software instructions that configure a particularCPU or other processor), including, but not limited to, one or moreapplication-specific integrated circuits (ASICs), standard integratedcircuits, controllers (e.g., by executing appropriate instructions, andincluding microcontrollers and/or embedded controllers),field-programmable gate arrays (FPGAs), complex programmable logicdevices (CPLDs), etc. Some or all of the modules, systems and datastructures may also be stored (e.g., as software instructions orstructured data) on a non-transitory computer-readable storage medium,such as a hard disk, a memory, a network, or a portable media article tobe read by an appropriate drive or via an appropriate connection. Thesystems, modules and data structures may also be transmitted asgenerated data signals (e.g., as part of a carrier wave or other analogor digital propagated signal) on a variety of computer-readabletransmission mediums, including wireless-based and wired/cable-basedmediums, and may take a variety of forms (e.g., as part of a single ormultiplexed analog signal, or as multiple discrete digital packets orframes). Such computer program products may also take other forms inother embodiments. Accordingly, the present invention may be practicedwith other computer system configurations.

FIGS. 4A-4C are a flow diagram of an example embodiment of a DistributedProgram Execution Service routine 400. The routine may be provided by,for example, execution of the DPE service modules 110 and 180 of FIGS.1A and 1B, respectively, the DPE system 340 of FIG. 3, and/or the DPEservice (not shown) that performs the activities described with respectto FIGS. 2A-2E, such as to manage distributed execution of programs andperform dynamic scaling of particular clusters, as well as to performother types of actions in some situations. In this illustratedembodiment, the routine 400 manages various aspects of use of a DPEservice that performs distributed execution of programs on behalf ofmultiple users.

In the illustrated embodiment, the routine begins at block 405, whereinformation related to distributed program execution is received. Theroutine continues to block 410 to determine the type of receivedinformation. If it is determined in block 410 that a request is receivedto initiate execution of a program, such as from a user, the routinecontinues to block 420 to execute an Obtain User-Specified ProgramExecution Configuration Information routine, with one example of such aroutine being described in greater detail with respect to FIG. 5. Afterthe program execution configuration information is obtained in block420, the routine continues to block 430 to execute a Perform DistributedProgram Execution routine, with one example of such a routine beingdescribed in greater detail with respect to FIG. 6. While the obtainingof program execution configuration information and performing ofdistributed program execution in blocks 420 and 430 are illustrated asbeing performed in a synchronous manner, such that the routine 400 doesnot perform other operations during those blocks, it will be appreciatedthat the routine may continue to perform other operations asynchronouslyand simultaneously with the actions of blocks 420 and 430, including tomonitor and modify ongoing program execution with respect to thoseblocks, as well as to perform other actions with respect to executingother programs for other users, as described elsewhere.

If it was instead determined in block 410 that the information receivedin block 405 was a request to register a program for later use, such asfrom a user, the routine instead continues to block 450 to storeinformation about the program, along with associated administrativeinformation (e.g., information about the user who submitted the program;information about execution of the program, such as how to separate theprogram into multiple execution jobs, a type of the program, etc.;optionally information about input data to be used during laterexecution of the program; optionally defined scaling criteria for lateruse in determining whether to dynamically scale a cluster being used fordistributed execution of the program, etc.). In some embodiments, suchprograms may be registered by users before they are used, such as toincrease the speed of executing the program when such a programexecution request is later received. Furthermore, in some embodiments,such a request to register a program may include a request to schedulelater execution of the program, such as at an indicated time or when oneor more indicated execution criteria are satisfied (e.g., whensufficient excess computing capacity of the DPE service is available toexecute the program at a low priority).

If it is instead determined in block 410 that the information receivedin block 405 is status information related to execution of one or moreexecution jobs as part of distributed execution of a program, such asmay be supplied by those execution job(s) or associated computingnode(s) performing the execution (e.g., after being previously requestedby the routine 400 as part of block 430), the routine continues to block455 to update status information that is maintained by the routine 400regarding the intermediate state of execution of that execution job andprogram. As discussed in greater detail elsewhere, the statusinformation may include information about particular operations thathave been initiated, completed, or are in progress, information aboutinput data that has been used by the execution, information about outputdata that has been generated by completion of some or all operations,information about partial intermediate data that reflects ongoingexecution of the execution job, etc.

If it is instead determined in block 410 that the information receivedin block 405 is an indication related to modifying the ongoingdistributed execution of one or more programs, such as from a user whoinitiated that execution and/or as previously specified by the routine400 with respect to blocks 441 or 490, the routine continues to block460 to receive information related to the execution modification. Inblock 462, the routine then determines whether the request relates to atemporary termination or other suspension of one or more currentlyexecuting execution jobs. If so, the routine continues to block 467 to,for each of one or more execution jobs to be suspended, obtaininformation about the current intermediate state of the partialexecution of the operations being performed for the execution job by itsassociated computing node and about the current data state for theexecution job on the associated computing node, and initiate the storageof the intermediate state information on one or more storage volumesremote from the computing node that will persistently store theinformation for later use. As previously discussed, in some embodimentsthe suspension may include temporarily terminating all execution jobscurrently being executed for a program, such as to temporarily suspendexecution of the program. In addition, if the computing node includesmanagement software to perform such intermediate state storage actionsor if the execution job itself is configured to save its ownintermediate state when instructed, the routine 400 may instead send amessage to the computing node and/or execution job to initiate suchstorage and then shut down. Furthermore, previously saved statusinformation about the execution job may in some situations be used todetermine what intermediate state information is be stored, and any newinformation obtained in block 467 (e.g., by communicating with thecomputing node and/or execution job) may be used to update the statusinformation for that execution job. After block 467, the routinecontinues to block 469 to optionally schedule a later resumption of theexecution of some or all of the execution jobs from the storedintermediate state. For example, in situations in which one or moreexecution jobs are being moved from a first group or one or morecomputing nodes to a second group or one or more other computing nodes,the later scheduled resumption of the execution of those execution jobsmay occur in a substantially immediate manner, such that a user may beunaware of the termination and resumption of execution or of themovement of the execution job between computing nodes. In addition, thescheduling of the later resumption may include initiating a new requestto be received in block 405 at the time of the later resumption thatwill prompt resumption activities for those execution jobs.

If it was instead determined in block 462 that the request was not toperform suspension of one or more execution jobs, the routine continuesto block 464 to determine whether the request is instead related toresuming execution of one or more previously suspended execution jobs.If so, the routine continues to block 465 to initiate the resumption ofexecution of each of one or more previously suspended execution jobs ona computing node, such as by using previously stored state informationabout the intermediate results from the previous partial execution ofthe execution job, or instead schedules a later such resumption of theexecution job if so indicated in the information received in block 460.

If it is instead determined in block 464 that the request is not relatedto resuming execution of previously suspended execution jobs, theroutine continues to perform blocks 470-480 related to dynamic scalingof a cluster, such as under control of the Cluster Modification Managermodule 347 of FIG. 3. In particular, in block 470, the routinedetermines if the request is to dynamically scale a cluster by reducingthe quantity of computing nodes in the cluster. If so, the routinecontinues to block 472 to perform the reduction by selecting one or morecomputing nodes and removing them from the cluster. In this illustratedembodiment, the routine selects to remove auxiliary nodes if they areavailable and to remove core computing nodes otherwise, but in otherembodiments may proceed in other manners (e.g., to not remove core nodesif sufficient auxiliary nodes are not available). In addition, in thisillustrated embodiment, the routine selects to remove temporarilyavailable nodes being used in the cluster if they are available, but inother embodiments may proceed in other manners (e.g., to not removetemporarily available nodes if sufficient other nodes are notavailable). In other embodiments, the request may include additionalinformation related to how to perform the reduction, such as a type ofauxiliary or core node to be selected for reduction, a resourceconfiguration type of computing node to be selected for reduction, anindication that computing nodes of a particular source are to beselected or not selected (e.g., an indication to perform the reductionby removing one or more computing nodes that were previously temporarilyadded to the cluster), an indication of a particular geographical and/ornetwork location of computing nodes that are to be selected, etc., andif so the routine may perform the reduction in accordance with theinstructions. In addition, in some embodiments, an intermediateexecution state may be saved for some or all nodes being removed (e.g.,in a manner similar to that described with respect to block 467), suchas to enable later completion of the execution jobs being performed onthose removed computing nodes by resuming from the intermediate state onother computing nodes of the cluster, although in other embodimentsthose execution jobs may instead be later completed by restarting theexecution jobs from the beginning on other computing nodes of thecluster.

If it is instead determined in block 470 that the request is not toreduce the quantity of computing nodes in a cluster, the routinecontinues instead to block 475 to determine if the request is todynamically scale a cluster by increasing the quantity of computingnodes in the cluster. If so, the routine continues to blocks 477-479 toperform the increase by selecting a type of computing node to add and byadding one or more computing nodes of that type to the cluster. Inparticular, in the illustrated embodiment, the routine selects in block477 whether to add core or auxiliary nodes to the cluster, such as basedon whether the adding of the nodes is intended to be a permanent ortemporary increase, respectively, and/or based on one or more otherfactors. In addition, in the illustrated embodiment, the routine selectsin block 477 whether to add any temporarily available node to thecluster, such as based on their current price and availability and/or onother factors (e.g., based on whether core or auxiliary nodes are beingadded). In some embodiments, the selection may include otherdeterminations and/or be made in other manners, such as to select aresource configuration type to use for the nodes being added, tootherwise select a particular source of the computing nodes to add basedon an associated fee and/or associated use conditions for nodes fromthat source, to select a particular geographical and/or network locationof computing nodes to be added, etc. In some embodiments, the requestmay include additional information related to how to perform theincrease, such as a type of auxiliary or core node to be selected, aresource configuration type of computing node to be selected, anindication of whether the increase is temporary or permanent, anindication of a particular source to use and optionally fee-relatedfactors to use in determining whether to add computing nodes of thatsource, etc., and if so the routine may perform the increase inaccordance with the instructions. In addition, in some embodiments, ifone of the added computing nodes begins to perform an execution job thatwas previously partially completed (e.g., by a computing node removed inblock 472), that computing node may obtain and use a previously savedintermediate execution state to enable completion of the execution jobby resuming from the intermediate state. If it is instead determined inblock 475 that the request is not to dynamically scale a cluster byincreasing the quantity of computing nodes in the cluster, the routinecontinues instead to block 480 to optionally perform one or more otherindicated actions related to dynamic scaling of one or more clusters,such as to otherwise initiate the removal of one or more computing nodesfrom a cluster that were previously temporarily added for an indicatedperiod of time or until other indicated conditions were satisfied.

If it instead determined in block 410 that the type of informationreceived in block 405 is an indication to dynamically monitor one ormore clusters (e.g., periodically or when triggered by a particularevent), such as to determine whether any defined scaling criteria forthe clusters are satisfied, and optionally with respect to the ongoingdistributed execution of a particular indicated program or instead withrespect to the ongoing distributed execution of some or all programsbeing executed by the DPE service, the routine continues to block 433.The routine retrieves information in block 433 related to any suchdefined scaling criteria for the one or more clusters being monitored,and then continues to block 435. In block 435, the routine gathersaggregate information regarding the usage of computing resources by theongoing distributed execution of one or more programs on the one or moreclusters, and optionally retrieves status information regarding thatongoing distributed execution of the one or more programs (e.g., statusinformation previously received and stored with respect to block 455,status information that is dynamically obtained by interacting with someor all computing nodes of each cluster performing the distributedexecution of one of the programs, etc.). As discussed elsewhere, theaggregate information regarding the usage of the computing resources maybe obtained in various manners, including by interacting with some orall computing nodes of a cluster performing the distributed execution ofa program to obtain information specific to those computing nodes, andthen aggregating the various node-specific information. After block 435,the routine continues to block 438 to determine if the gatheredaggregate computing resource usage information for the one or moreprograms indicates that any defined scaling criteria are satisfied, andto determine if any other current information indicates that any definedscaling criteria are satisfied, such as current price and availabilityof temporarily available computing nodes with respect to any definedscaling criteria that correspond to requests to add or remove suchcomputing nodes at indicated price thresholds. In some embodiments, thegathered aggregate computing resource usage information may further beused in other manners, such as to determine if any computing resourcesof the DPE service are being over-utilized. Such over-utilization ofcomputing resources by the ongoing distributed execution of the one ormore programs may, for example, create a bottleneck for at least one ofthose one or more programs and/or for one or more other programs, and/ormay be based on the one or more programs using more computing resourcesthan are allocated to those programs or that are otherwise expected tobe used by those one or more programs. After block 438, the routinecontinues to block 441 to, if any defined scaling criteria aredetermined to be satisfied (and/or if any over-utilization of computingresources is detected), take corresponding actions. If one or moredefined scaling criteria for a particular cluster are determined to besatisfied, the actions in block 441 include initiating a request toperform corresponding dynamic scaling of the cluster, as described ingreater details with respect to blocks 470-479, and in FIG. 8 withrespect to requests to add or remove temporarily available computingnodes. In some embodiments and situations, such actions may alsoinclude, for example, initiating throttling of the use of the computingresources by at least one of the one or more programs, and/or mayinclude modifying the amount of computing resources available to thoseone or more programs (e.g., to increase an allocation of computingresources available to those one or more programs, such that the currentcomputing resource usage does not exceed the increased allocation),whether on the current computing nodes of one or more clusters or bymodifying the computing nodes of the one or more clusters.

If it instead determined in block 410 that the type of informationreceived in block 405 is another type of request, the routine continuesinstead to block 485 to perform the other indicated request asappropriate. Such other requests may include, for example,administrative operations with respect to users, such as usersregistering with the DPE service, users checking account information orother information of the DPE service, users tracking ongoing executionof their programs (e.g., by receiving some or all information about theintermediate state or other status of one or more of the execution jobsfor the program), users providing payment with respect to currently orpreviously requested program executions or other provided functionality,etc.

After blocks 430, 441, 450, 455, 465, 469, 472, 475, 479, 480, or 485,the routine continues to block 490 to optionally perform one or moreother tasks. Such other tasks may include, for example, one or more ofthe following: the DPE service automatically initiating the monitoringof the execution of various programs, such as by sending requests forstatus information to computing nodes (e.g., periodically, whenindicated execution criteria are satisfied, etc.) and/or initiating themonitoring described with respect to blocks 433-441; automaticallydetermining to temporarily terminate execution of execution jobs and/orto resume previously terminated execution of execution jobs in variouscircumstances, such as at scheduled times, to accommodate movingexecution jobs between computing nodes, to maintain requested QoS levelsfor execution of a program (e.g., by maintaining execution of executionjobs within a maximum proximity to each other and/or to input data beingused by the execution jobs, by maintaining a minimum proximity ofexecution to each other to enhance availability and reliability in theevent of failure of multiple computing nodes in a single geographicallocation, etc.), to manage under-utilization and over-utilization ofcomputing nodes (e.g., by modifying clusters and/or computing resourcesavailable to clusters), etc. After block 490, the routine continues toblock 495 to determine whether to continue, such as until an explicitindication to terminate is received (e.g., from an operator of the DPEservice). If it is determined to continue, the routine returns to block405, and if not continues to block 499 and ends.

FIG. 5 is a flow diagram of an example embodiment of an ObtainUser-Specified Program Execution Configuration Information routine 500.The routine may be provided by, for example, execution of the UserInteraction Manager module 342 of the DPE system 340 of FIG. 3, orotherwise by the DPE service modules 110 and 180 of FIGS. 1A and 1B,respectively, and may be initiated by block 420 of FIG. 4. The routine500 interacts with users to determine configuration information to beused to execute indicated programs in a distributed manner on multiplecomputing nodes, including to determine various execution configurationparameters and/or defined scaling criteria related to later dynamicscaling to be performed.

The routine begins in block 505, where an indication is received onbehalf of a user regarding configuration information to be specified bythe user regarding distributed execution of an indicated program. If itis determined in block 505 that the received indication is received viaan API of the DPE service, the routine continues to block 510 to obtainand stored program execution configuration information via one or moreAPI function calls, and to optionally store and aggregate suchinformation over multiple API function calls. After the various programexecution configuration information has been obtained, and aninstruction is received via the API to initiate distributed execution ofthe indicated program in accordance with the specified configurationinformation, the routine in block 510 continues to block 599 andreturns, including providing an indication of that specifiedconfiguration information and initiation instruction.

If it is instead determined in block 505 that the received indication isnot received via an API, such as via a user interaction with apreviously provided GUI of the DPE service or instead via a user requestto obtain access to such a GUI, the routine continues instead to block515 to provide (or update as appropriate) a GUI of the DPE service fordisplay to the user, with the initial GUI screen(s) being configured toobtain information about an indicated program to execute and aboutvarious program execution configuration information. In block 520, theroutine then receives information from one or more interactions of theuser with the displayed GUI that indicates the program to be executedand optionally various other user-specified configuration information.While not illustrated here, in some embodiments the routine may furtherprovide information to at least some users about recommended definedscaling criteria for dynamic scaling (e.g., based on a type of theprogram to be performed, on indicated objectives of the user, etc.)and/or about other recommended execution configuration parameters fordistributed execution of the indicated program. As described in greaterdetail elsewhere, the information about the recommended executionconfiguration parameters may be used in various manners and times invarious embodiments, including for display to the user (e.g., for use asmodifiable default values for the execution configuration parameters,for use in providing a warning about other user-specified executionconfiguration parameters, etc.). In addition, the recommended executionconfiguration parameters may be determined in various manners in variousembodiments, as discussed elsewhere. After the various configurationinformation has been provided, the routine receives instructions fromthe user in block 540 to initiate the distributed execution of theindicated program in accordance with the specified configurationinformation, and continues to block 599 and returns, including providingan indication of that specified configuration information and initiationinstruction.

While not illustrated here, it will be appreciated that in someembodiments a user may begin the interactions described with respect toFIG. 5, and then decide to explicitly cancel the interactions orotherwise fail to complete the interactions. In such situations, theroutine may exit without providing an indication of the programexecution initiation instruction (and optionally without anyconfiguration information specified by the user before thecancellation/completion failure), and the routine 400 in FIG. 4 mayinstead determine to skip block 430 due to the lack of the programexecution initiation instruction.

FIG. 6 is a flow diagram of an example embodiment of a PerformDistributed Program Execution routine 600. The routine may be providedby, for example, execution of the DPE service modules 110 and 180 ofFIGS. 1A and 1B, respectively, and/or the DPE system 340 of FIG. 3, andmay be initiated by block 430 of FIG. 4. The routine 600 performsoperations to execute indicated programs in a distributed manner onmultiple computing nodes in accordance with specified configurationinformation.

The routine begins in block 610, where it receives program executionconfiguration information for use in executing an indicated program,such as from the output of routine 500 of FIG. 5. After block 610, theroutine continues to block 615 to determine a quantity of computingnodes to be used in a cluster for the program execution, such as isspecified in the received execution configuration information, orotherwise automatically determined (e.g., based on preferred or defaultconfiguration information, a number of computing nodes that arecurrently available from the DPE service, a number of computing nodes tocorrespond to a number of execution jobs into which the programexecution will be separated, an amount of fees paid by a user on whosebehalf the request is made, etc.). In addition, the routine in block 615may determine other information regarding the initiation of thedistributed program execution, such as whether some/all of the clustercomputing nodes are to be used as core nodes and/or as auxiliary nodes,one or more resource configuration types to use for the initialcomputing nodes, a particular source of the computing nodes to be usedbased on an associated fee and/or associated use conditions for nodesfrom that source (e.g., whether any temporarily available computingnodes are currently available within a specified maximum pricethreshold, such as whether a bid request for such temporarily availablecomputing nodes is currently satisfied), a particular geographicaland/or network location of computing nodes to be used, etc. In block620, the routine separates the program execution and any received inputdata into multiple execution jobs that each involves one or moreoperations to be performed. The actions in blocks 615 and 620 may beperformed in various manners in various embodiments, as discussed ingreater detail elsewhere. For example, the program execution separationmay be performed in a manner specified in the program executionconfiguration information, in an automated manner based at least in parton the design of the program, etc. After block 620, the routinecontinues to block 625 to select the determined quantity of computingnodes for use in distributed execution of the program based on theinformation determined in block 615, and in block 630 initiates theexecution of at least some of the execution jobs on the selected nodes.As discussed in greater detail elsewhere, in some embodiments a user mayspecify a variety of other types of information, such as may be receivedas part of the configuration information received in block 610 and usedas part of the actions in blocks 615-630.

After block 630, the routine continues to block 635 to perform variouscontinuing activities over time as part of performing the distributedexecution of the program on the cluster of computing nodes. Suchcontinuing activities may include determining at a later time that thereare more execution jobs available to be executed at that time, andinitiating execution of one or more of those execution jobs on currentlyavailable computing nodes of the cluster (e.g., based on the computingnodes of the cluster after any prior dynamic scaling has been performed,and which are not otherwise currently occupied in the distributedexecution of the program), such as if those execution jobs had beenwaiting for output data from one or more of the completed executionjobs, or instead if there had initially been more execution jobs thanavailable computing nodes. Such continuing activities may also includewaiting for execution jobs to complete and to optionally providecorresponding output data, such as may be used as input data to otherexecution jobs and/or may be used as part or all of the final resultsfor the execution of the program. In some embodiments, the computingnodes may supply such output information back to the routine 400, suchas for storage in a long-term storage location of the DPE service, whilein other embodiments the output results may instead be stored on thecomputing nodes (e.g., as part of a distributed storage system onmultiple core computing nodes), and/or stored by the computing nodes onone or more long-term storage locations remote from the computing nodes.In addition, the continuing activities may include dynamically scalingthe cluster or performing other cluster modifications, such as inresponse to user instructions or satisfaction of defined scalingcriteria, as described in greater detail with respect to blocks 460-480of FIG. 4C, and in FIG. 8 with respect to any temporarily availablecomputing nodes being used or considered for use. In the illustratedembodiment, blocks 630-655 are illustrated as being performed in asynchronous manner in which the routine 600 waits for execution jobs tocomplete before performing other actions. It will be appreciated that inother embodiments the routine 600 may operate in other manners, such asin an asynchronous manner to perform other actions with respect toexecuting the current program and/or to executing other programs forother users. In addition, it will be appreciated that in some situationserrors may occur that cause one or more execution jobs to fail tocomplete, such as due to problems with the computing node on which theexecution job is being performed, due to a network connection with thecomputing node, due to an error in the software corresponding toperforming the execution job, due to problems with input data to be usedfor the performance of the execution job, etc. In such situations, theroutine may take various actions to correct the errors (e.g., bytransferring the execution job to another computing node), but if anirreversible error occurs, the routine may terminate the furtherdistributed execution of the program or may instead attempt to completeas much of the distributed execution of the program as possible andprovide incomplete final results along with an indication that theprogram executed is completed with errors.

After all of the execution jobs have completed in block 635, the routinecontinues to block 685 to produce final execution results for theprogram, and to optionally store those final results and/or provide themto the user. The final execution results may be generated or otherwiseproduced in various manners in various embodiments, such as by combiningoutput results from multiple execution jobs in various ways, selectingthe output results from a single execution job to use, etc. After block685, the routine continues to block 699 and returns.

FIG. 7 is a flow diagram of an example of a Computing Node Managerroutine 700. The routine may be provided by, for example, execution ofmodule 179 of FIG. 1B, or of other management software executing oncomputing nodes 120 of FIG. 1A, computing systems 182 of FIG. 1B,computing nodes 360 of FIG. 3, etc. In the illustrated embodiment, themanagement software on a computing node manages some aspects of theexecution of execution jobs on the computing node, while in otherembodiments some or all of the functionality described with respect toFIG. 7 may instead be performed by the routine 400 of FIGS. 4A-4C, orinstead by an execution job that is being executed on the computingnode.

In the illustrated embodiment, the routine begins at block 705, where anindication is received related to the execution of one or more executionjobs on the computing node. The routine continues to block 710 todetermine whether the indication is to initiate execution of anindicated execution job. If so, the routine continues to block 713 toobtain the input data to be used for the indicated execution job and tostore it locally, and to determine the multiple operations to beperformed for the execution job (e.g., based on the information receivedabout the execution job in block 705 or that is otherwise retrieved froman external source). The input data to be used may be obtained invarious manners, such as by being received in block 705, already beingavailable on local storage (e.g., if the computing node is a core nodewhose local portion of the distributed storage system stores that inputdata), being retrieved from another computing node that is a corecomputing node (e.g., if the current computing node is an auxiliary nodeor is a core node whose local portion of the distributed storage systemdoes not store that input data), etc. After block 713, the routinecontinues to block 715 to select the next operation to be performed,beginning with the first. Blocks 715-740 are illustrated in this exampleembodiment as being performed in a serial manner, such that eachoperation is individually performed in a synchronous manner before thenext operation is initiated, although it will be appreciated that inother embodiments the operations may be performed in other manners, suchas to execute some or all of the operations in parallel and/or in anasynchronous manner.

After block 715, the routine continues to block 720 to send a statusmessage to the routine 400 indicating that the execution of the selectedoperation has begun, and in block 725 initiates the performance of theoperation. In block 730, the routine waits for the completion of theperformance of the operation, and optionally locally stores anyintermediate results produced by the operation performance. In block735, the routine then sends another status message to the routine 400indicating that the performance of the operation is completed. In block740, the routine then determines whether there are more operations to beperformed, and if so returns to block 715.

If was instead determined in block 710 that the received indication inblock 705 was not to execute an execution job, the routine continues toblock 745 to determine whether the received indication in block 705 isto perform one or more activities related to participation in adistributed storage system, although such activities may be performedonly for core nodes in at least some embodiments. If so, the routinecontinues to block 750 to take actions to perform one or more suchactivities. The actions may include storing indicated data in a localportion of the distributed storage system and/or retrieving andproviding such locally stored data to a requester (e.g., to anothercomputing node with respect to block 713 of a routine performed for thatother computing node), and may further include taking various activitiesto establish the participation of the computing node in the distributedstorage system when the use of the cluster is first initiated.

If was instead determined in block 745 that the received indication inblock 705 was not to participate in the distributed storage system forthe cluster, the routine continues instead to block 785 to perform oneor more other indicated operations as appropriate. For example, theinformation received in block 705 may be a request for statusinformation from the routine 400, and if so block 785 may obtain andprovide that status information to the routine 400. Alternatively, theinformation in block 705 may be an indication to terminate execution ofthe execution job, and the actions performed in block 785 may includecorresponding actions (e.g., to clear intermediate state informationthat was temporarily stored on the computing node, such as after thatinformation has been persistently stored elsewhere). In addition,actions performed with respect to block 785 may include a request fromthe routine 400 to initiate such persistent storage of intermediatestate from partial execution of operations for a currently executingexecution job, and if so the actions in block 785 may include performingsuch actions (whether immediately or instead after a short time to allowcompletion of performance of one or more operations). Similarly, actionsperformed with respect to block 785 may include a request from theroutine 400 to throttle or otherwise modify computing resource usage bythe computing node, and if so the actions in block 785 may includeperforming such actions. Furthermore, in some embodiments andsituations, different execution jobs may coordinate in various manners,such as by exchanging information of interest, and the information inblock 705 may be such information from another execution job of theprogram or a request from such other execution job for information fromthe current execution job.

After blocks 750 or 785, or if it is instead determined in block 740that there are no more operations to be performed, the routine continuesto block 790 to optionally perform one or more other tasks, such as tocoordinate with other execution jobs of the program if appropriate, totake administrative actions as part of executing the execution job, etc.After block 790, the routine continues to block 795 to determine whetherto continue, such as until an explicit indication to terminate isreceived (e.g., from the routine 400 or from a human operator of the DPEservice). If it is determined to continue, the routine returns to block705, and if not continues to block 799 and ends.

FIG. 8 is a flow diagram of an example embodiment of a Capacity UsageManager routine 800. The routine may be provided by, for example,execution of one of the other modules 349 of FIG. 3 and/or by executionof the DPE service modules 110 and 180 of FIGS. 1A and 1B, respectively.In the illustrated embodiment, the routine manages bids and otherrequests to use temporarily available excess capacity computing nodes,such as may be initiated by client instructions and/or satisfaction ofdefined scaling criteria for clusters, although in the other embodimentsmay instead by performed as part of routine 400 of FIGS. 4A-4C. Inaddition, in the illustrated embodiment the routine 800 is performedperiodically (e.g., as initiated by block 490 of routine 400), althoughin other embodiments may be executed in a continuous manner.

In the illustrated embodiment, upon invocation of the routine 800, arequest, information and/or other indication related to excess capacityuse is provided to the routine. At block 805, the routine determineswhether an indication is received to add one or more computing nodes toavailable excess program execution capacity, such as based oninformation obtained with respect to monitoring activities that areperformed (e.g., with respect to block 490 of routine 400). If so, theroutine continues to block 810 to add the indicated computing node(s) tothe available program execution capacity, such as to a correspondingpool of temporarily available computing nodes. In some embodiments,multiple pools may be provided, such that, for example, computing nodesof different resource configuration types may be available as excesscapacity nodes and may be grouped into distinct excess capacity pools.If it is instead determined at block 805 that an indication to addavailable excess program execution capacity is not received, the routinecontinues to block 815 to determine whether an indication is received toremove one or more computing nodes from the available excess programexecution capacity, and if so, the routine continues to block 820 toremove the computing node(s) from the available excess program executioncapacity, such that those computing nodes (or an equivalent amount ofprogram execution capacity) is no longer available. As discussedelsewhere, such removal may include removing computing nodes from clientclusters and terminating execution of execution jobs on those computingnodes.

If it is instead determined in block 815 that an indication to removeavailable program execution capacity is not received, the routinecontinues instead to block 825 to determine whether a request isreceived to use one or more available excess capacity computing nodes toperform distributed execution of a program as part of a client cluster(or a previously scheduled request for such use has been triggered). Ifso, the routine continues to block 830 to optionally obtain or receiveinformation related to the requested use, such as indications of anamount and/or type of program execution capacity requested (e.g.,including minimum and maximum amounts), a bid amount (e.g., includingminimum and maximum bid amounts), an expiration time, etc. In addition,in some embodiments, the routine may provide feedback of one or moretypes, such as indicating likely characteristics of the requested use(e.g., a likely time that the request will be satisfied, a likelyduration of available use, etc.) and/or optional other information(e.g., suggested configurations). After block 830, the routine continuesto block 835 to add the request to a group of current requests forexecution on excess program execution capacity. In other embodiments,rather than add the request to a group of other requests, the routinemay instead attempt to immediately satisfy the request, such as bydetermining whether there is available excess program execution capacityto satisfy the request at that immediate time, and initiating the use ifso, or failing if not.

If it is instead determined at block 825 that a request to use availableexcess program execution capacity is not received, the routine continuesto block 840 to determine whether a request to remove an indicatedprevious request is received (e.g., a indicated request that waspreviously added to the group of requests at block 835), such as arequest from a client to cancel a current request for execution and/or areceived indication that the request has otherwise expired, etc. If so,the routine continues to block 845 to terminate any current useassociated with the indicated request, and then continues to block 850to remove the indicated request from the group of requests, such thatthe indicated request will not subsequently be considered forsatisfaction on excess program execution capacity.

After blocks 810, 820, 835 and 850, or if it was instead determined atblock 840 that a request to remove an indicated request is not received,the routine continues to block 855 to determine whether to managecurrent requests for excess program execution capacity. For example, insome embodiments, an indication may have been passed into the routine800 to manage the requests and/or may have been provided based upon achange in the available program execution capacity (e.g., by adding toor removing from the available program execution capacity in blocks 810and 820, respectively), and/or a change in the current request forexecution (e.g., by adding or removing requests in blocks 835 and 850,respectively), etc. If so, the routine continues to block 860 todetermine an amount of currently available excess program executioncapacity, such as including an amount of program execution capacityadded and/or removed from the available program execution capacity atblocks 810 and 820, respectively. After block 860, the routine continuesto block 865 to allocate currently available excess capacity computingnodes for use with one or more of the requests in the group based atleast in part on priorities associated with the requests, such as tomake the allocated excess capacity computing nodes available for use aspart of the indicated cluster. For example, in some embodiments, theroutine attempts to allocate to each request in the group a specifiedquantity of requested excess capacity computing nodes. In some suchembodiments, if there is sufficient currently available excess programexecution capacity to satisfy all the current requests in the group ofrequests, then each of the requests will be allocated a requested amountof the excess program execution capacity. In cases where there is notsufficient currently available excess program execution capacity tosatisfy all the current requests, the routine prioritizes the requestsand attempts to satisfy higher priority requests before lower priorityrequests. In various embodiments, priority may be based on one or moreof a bid amount, a time at which the request was received, etc. Invarious embodiments, the routine may determine which of the currentlyavailable excess program execution capacity to allocate for use by therequests in various ways, such as described elsewhere.

After block 865, the routine continues to block 870 to terminate anypreviously initiated ongoing use associated with a request that was notallocated for further current use by the request in block 865, such asto reflect that the request has expired or was not sufficiently highpriority to result in current ongoing satisfaction. After block 870, theroutine continues to block 880 to optionally perform housekeepingoperations. For example, in some embodiments, the routine may detect andindicate if there are any requests for execution in the group ofrequests that are expired or otherwise should be removed.

If it is instead determined at block 855 not to manage current requestsfor execution on excess program execution capacity, the routinecontinues to block 895 to optionally perform one or more other indicatedoperations as appropriate. After blocks 880 and 895, the routinecontinues to block 899 and returns.

It will be appreciated that in some embodiments the functionalityprovided by the routines discussed above may be provided in alternativeways, such as being split among more routines or consolidated into fewerroutines. Similarly, in some embodiments illustrated routines mayprovide more or less functionality than is described, such as when otherillustrated routines instead lack or include such functionalityrespectively, or when the amount of functionality that is provided isaltered. In addition, while various operations may be illustrated asbeing performed in a particular manner (e.g., in serial or in parallel)and/or in a particular order, those skilled in the art will appreciatethat in other embodiments the operations may be performed in otherorders and in other manners. Those skilled in the art will alsoappreciate that the data structures discussed above may be structured indifferent manners, such as by having a single data structure split intomultiple data structures or by having multiple data structuresconsolidated into a single data structure. Similarly, in someembodiments illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered.

From the foregoing it will be appreciated that, although specificembodiments have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the invention. Accordingly, the invention is not limited exceptas by the appended claims and the elements recited therein. In addition,while certain aspects of the invention are presented below in certainclaim forms, the inventors contemplate the various aspects of theinvention in any available claim form. For example, while only someaspects of the invention may currently be recited as being embodied in acomputer-readable medium, other aspects may likewise be so embodied.

1-12. (canceled)
 13. A non-transitory computer-readable medium whosecontents configure a computing system to perform a method, the methodcomprising: initiating execution of an indicated program in adistributed manner on a cluster of multiple computing nodes at a firsttime, the initiating of the execution including attempting to executeone or more of a plurality of jobs of the indicated program on each ofthe multiple computing nodes, the execution of each of at least some ofthe plurality of jobs including use of indicated input data; at a secondtime subsequent to the first time and while the execution of theindicated program is ongoing, determining to modify the cluster duringthe ongoing execution of the indicated program based at least in part onuse of one or more computing nodes that are temporarily available untila distinct use of a higher priority occurs for the one or more computingnodes, the cluster including at the second time a first group of themultiple computing nodes that each are part of distributed storage forthe indicated input data; and automatically initiating a change in themultiple computing nodes of the cluster while the execution of theindicated program is ongoing in response to the determining to modifythe cluster, the initiating of the change including selecting for theinitiated change a second group corresponding to computing nodes of thecluster that are not part of the distributed storage and includingperforming a modification to the second group corresponding to the useof the one or more temporarily available computing nodes, the automaticinitiating of the change being performed by the configured computingsystem.
 14. The non-transitory computer-readable medium of claim 13wherein the configured computing system is part of a distributed programexecution service that provides a plurality of computing nodes ofmultiple types that are configurable to perform distributed execution ofprograms for clients, wherein the initiating of the execution of theindicated program is performed for a first client, wherein the multiplecomputing nodes of the cluster are a subset of the plurality ofcomputing nodes that are provided for use on behalf of the first client,wherein the computing nodes of the first group that are part of thedistributed storage at the second time include multiple core computingnodes that are each part of a distributed storage system for the firstclient, wherein the cluster further includes at one or more times one ormore auxiliary computing nodes in the second group that are not part ofthe distributed storage system for the first client, and wherein theautomatic initiating of the change includes automatically modifying aquantity of the auxiliary computing nodes in the second group of thecluster.
 15. The non-transitory computer-readable medium of claim 14wherein the one or more temporarily available computing nodes are of oneof the multiple types of computing nodes, wherein the one type includescomputing nodes that are already allocated for a first use distinct fromuse in the cluster but that are not being currently used for thedistinct first use, wherein the one or more auxiliary computing nodes inthe second group at the one or more times include the one or moretemporarily available computing nodes in the second group at the secondtime, wherein the one or more temporarily available computing nodes areadded to the second group at an earlier time prior to the second timebased on an indicated instruction and on availability of the one or moretemporarily available computing nodes at the earlier time, wherein themultiple core computing nodes of the first group are of another of themultiple types of computing nodes that is distinct from the one type andthat includes computing nodes that are not already allocated for anyother uses distinct from use in the cluster, and wherein the performedmodification to the second group includes removing at least one of theone or more temporarily available computing nodes from the cluster basedon an indication that the removed computing nodes are to be used for thedistinct first use.
 16. The non-transitory computer-readable medium ofclaim 14 wherein the one or more temporarily available computing nodesare of one of the multiple types of computing nodes, wherein the onetype includes computing nodes that are already allocated for a first usedistinct from use in the cluster but that are not currently being usedfor the distinct first use, wherein the multiple core computing nodes ofthe first group are of another of the multiple types of computing nodesthat is distinct from the one type and that includes computing nodesthat are not already allocated for any other uses distinct from use inthe cluster, wherein the cluster does not include any computing nodes ofthe second group at the second time, and wherein the performedmodification to the second group includes creating the second group atthe second time by adding the one or more temporarily availablecomputing nodes to the cluster for use as at least part of the secondgroup.
 17. The non-transitory computer-readable medium of claim 14wherein the one or more temporarily available computing nodes are of oneof the multiple types of computing nodes, wherein the one type includescomputing nodes that are already allocated for a first use distinct fromuse in the cluster but that are not currently being used for thedistinct first use, wherein the multiple core computing nodes of thefirst group are of another of the multiple types of computing nodes thatis distinct from the one type and that includes computing nodes that arenot already allocated for any other uses distinct from use in thecluster, wherein the one or more auxiliary computing nodes in the secondgroup at the one or more times include one or more computing nodes ofthe another type in the second group at the second time, and wherein theperformed modification to the second group includes adding the one ormore temporarily available computing nodes to the second group of thecluster at the second time.
 18. The non-transitory computer-readablemedium of claim 14 wherein the one or more temporarily availablecomputing nodes are of one of the multiple types of computing nodes,wherein the one type includes computing nodes that are already allocatedfor a first use distinct from use in the cluster but that are notcurrently being used for the distinct first use, wherein the multiplecore computing nodes of the first group at the second time are ofanother of the multiple types of computing nodes that is distinct fromthe one type and that includes computing nodes that are not alreadyallocated for any other uses distinct from use in the cluster, andwherein the initiating of the change further includes adding at leastone temporarily available computing node of the one type to the firstgroup of the cluster at the second time.
 19. The non-transitorycomputer-readable medium of claim 14 wherein the one or more temporarilyavailable computing nodes are of one of the multiple types of computingnodes, wherein the one type includes computing nodes that are alreadyallocated for a first use distinct from use in the cluster but that arenot currently being used for the distinct first use, wherein themultiple core computing nodes of the first group at the second timeinclude the one or more temporarily available computing nodes of the onetype and include one or more other computing nodes of another of themultiple types of computing nodes that is distinct from the one type andthat includes computing nodes that are not already allocated for anyother uses distinct from use in the cluster, and wherein the performedmodification to the second group includes moving at least one of the oneor more temporarily available computing nodes from the first group ofthe cluster to the second group of the cluster in preparation for futureuse of the moved computing nodes for the distinct first use.
 20. Thenon-transitory computer-readable medium of claim 13 wherein theinitiating of the execution of the indicated program on the cluster ofmultiple computing nodes at the first time includes determining a subsetof a first quantity of the plurality of jobs whose execution is to beinitiated at the first time, wherein the first quantity of the jobs ofthe determined subset is based at least in part on a quantity of themultiple computing nodes of the cluster at the first time, and whereinthe automatic initiating of the change includes automatically increasingthe quantity of the multiple computing nodes in the cluster at thesecond time and includes initiating execution of a second quantity ofthe plurality of jobs on the multiple computing nodes in the clusterafter the second time, the second quantity being larger than the firstquantity based at least in part on the increasing of the quantity of themultiple computing nodes in the cluster at the second time.
 21. Thenon-transitory computer-readable medium of claim 13 wherein theinitiating of the execution of the indicated program on the cluster ofmultiple computing nodes at the first time includes determining a subsetof a first quantity of the plurality of jobs whose execution is to beinitiated at the first time, wherein the first quantity of the jobs ofthe determined subset is based at least in part on a quantity of themultiple computing nodes of the cluster at the first time, and whereinthe automatic initiating of the change includes automatically reducingthe quantity of the multiple computing nodes in the cluster at thesecond time and includes initiating execution of a second quantity ofthe plurality of jobs on the multiple computing nodes in the clusterafter the second time, the second quantity being smaller than the firstquantity based at least in part on the reducing of the quantity of themultiple computing nodes in the cluster at the second time.
 22. Thenon-transitory computer-readable medium of claim 13 wherein the multiplecomputing nodes of the first group at the second time provide thedistributed storage by each storing a portion of the indicated inputdata on local storage of that computing node, and wherein the methodfurther comprises, at a third time subsequent to the second time andwhile the execution of the indicated program is ongoing, and in responseto determining that one of the computing nodes of the first group hasbecome unavailable for use in the cluster, automatically replacing theone unavailable computing node in the cluster with a differentreplacement computing node that is added to the first group and isprovided with the portion of the indicated input data that was stored onthe local storage of the one unavailable computing node.
 23. Thenon-transitory computer-readable medium of claim 22 wherein the clusterincludes at the third time one or more computing nodes of the secondgroup that are not part of the distributed storage, and wherein themethod further comprises, at the third time and while the execution ofthe indicated program is ongoing, and in response to determining that asecond of the multiple computing nodes of the cluster has becomeunavailable for use in the cluster, the second unavailable computingnode being one of the computing nodes of the second group, automaticallydetermining not to replace the second unavailable computing node in thecluster with a different replacement computing node based at least inpart on the second computing node being part of the second group, theautomatic replacing of the one unavailable computing node of the firstgroup with a different replacement computing node being based at leastin part on the one unavailable computing node being part of the firstgroup.
 24. The non-transitory computer-readable medium of claim 13wherein the one or more temporarily available computing nodes are partof a pool of computing nodes that have an associated price of use thatvaries at differing times, wherein the associated price of use at thesecond time of the computing nodes of the pool is a first price, andwherein the determining to modify the cluster at the second time basedat least in part on the use of the one or more temporarily availablecomputing nodes includes one of initiating the use of the one or moretemporarily available computing nodes at the second time based on thefirst price being below a threshold and of terminating the use of theone or more temporarily available computing nodes at the second timebased on the first price being above a threshold.
 25. The non-transitorycomputer-readable medium of claim 24 wherein the method furthercomprises automatically monitoring the associated price of use of thecomputing nodes of the pool during the execution of the indicatedprogram, and wherein the determining to modify the cluster at the secondtime is initiated based on one or more defined criteria associated withthe indicated program being determined to be satisfied at the secondtime based on the associated price of use at the second time of thecomputing nodes of the pool, the one or more defined criteria beingdetermined to be satisfied at the second time based at least in part onthe monitoring.
 26. The non-transitory computer-readable medium of claim24 wherein the execution of the indicated program is initiated on behalfof a client, and wherein the method further comprises automaticallymonitoring the associated price of use of the computing nodes of thepool during the execution of the indicated program, and providinginformation to the client during the execution of the indicated programregarding the monitored price of use of the computing nodes of the pool.27. The non-transitory computer-readable medium of claim 13 wherein theperforming of the modification to the second group includes initiatingthe use of the one or more temporarily available computing nodes as partof the second group, and wherein the use of each of the one or moretemporarily available computing nodes as part of the second groupcontinues unless the distinct higher priority use arises for thetemporarily available computing node, at least one of the one or moretemporarily available computing nodes having continuing use as part ofthe second group until use of the cluster ends.
 28. The non-transitorycomputer-readable medium of claim 13 wherein the computer-readablemedium is a memory of the configured computing system that stores thecontents, and wherein the contents are instructions that when executedprogram the computing system to perform the method.
 29. A computingsystem, comprising: one or more processors; and one or more modules of adistributed execution service that are configured to, when executed byat least one of the one or more processors, manage distributed executionof programs for clients by, for each of one or more of the clients:receiving information from the client regarding execution of anindicated program in a distributed manner on a cluster of multiplecomputing nodes, the execution of the indicated program in thedistributed manner including executing a plurality of jobs of theindicated program; initiating the execution of the indicated program inthe distributed manner on the cluster of multiple computing nodes at afirst time, the initiating of the execution including attempting toexecute one or more of the plurality of jobs of the indicated program oneach of one or more of the multiple computing nodes; at a second timesubsequent to the first time and while the execution of the indicatedprogram is ongoing, automatically determining to modify a quantity atthe second time of first computing nodes in the cluster that eachexecute one or more of the plurality of jobs of the indicated programwithout being part of distributed storage for information used for theongoing execution of the indicated program, the first computing nodesbeing distinct from at least some other computing nodes in the clusterat the second time that each are part of the distributed storage andeach are of a first type that enables the at least some computing nodesto be permanently assigned to the cluster until the cluster terminates;and automatically initiating a change in the quantity of the firstcomputing nodes of the cluster while the execution of the indicatedprogram is ongoing and in response to the determining to modify thequantity, the initiated change in the quantity of the first computingnodes corresponding to use of one or more computing nodes of a secondtype that are temporarily available for use in the cluster until adistinct use occurs for the one or more computing nodes.
 30. Thecomputing system of claim 29 wherein the distributed execution serviceis a distributed program execution service that provides a plurality ofcomputing nodes configurable to perform distributed execution ofprograms for the clients of the distributed program execution service,wherein the multiple computing nodes of the cluster for a first of theone or more clients are a subset of the plurality of computing nodes,wherein the at least some computing nodes in the cluster of the firstclient at the second time that are part of the distributed storage forthe first client includes a first group of multiple core computing nodesthat are each part of a distributed storage system for the first client,wherein the cluster of the first client further includes at one or moretimes one or more of the first computing nodes in a second group of oneor more other auxiliary computing nodes that are not part of thedistributed storage system, and wherein the automatic initiating of thechange includes automatically modifying a quantity of the auxiliarycomputing nodes in the second group of the cluster.
 31. The computingsystem of claim 30 wherein the information received from the firstclient is configuration information for the execution of the indicatedprogram for the first client that includes an indication of a quantityof the multiple computing nodes in the cluster for the first client atthe first time and that includes one or more criteria that is used inthe automatic determining at the second time to modify the quantity ofthe first computing nodes in the cluster for the first client, whereinthe one or more computing nodes of the cluster for the first client onwhich one or more jobs are attempted to be executed at the first timeinclude the one or more core computing nodes and include the one or moreother auxiliary computing nodes, wherein the one or more temporarilyavailable computing nodes for use in the cluster for the first clientare already allocated for a first use distinct from use in the clusterfor the first client but are not being currently used for the distinctfirst use, wherein the one or more auxiliary computing nodes in thesecond group at the one or more times include the one or moretemporarily available computing nodes in the second group at the secondtime, and wherein the initiated change in the quantity of the firstcomputing nodes corresponding to the use of the one or more temporarilyavailable computing nodes includes removing at least one of the one ormore temporarily available computing nodes from the cluster based on anindication that the removed computing nodes are to be used for thedistinct first use.
 32. The computing system of claim 29 wherein the oneor more modules each include software instructions for execution by thecomputing system.
 33. (canceled)